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The key aim of this Event is to provide a Forum for the user, supplier and scientific research 
communities to meet and exchange experiences, ideas and plans in the wide area of Culture & 
Technology. Participants receive up to date news on new EC and international arts computing & 
telecommunications Initiatives as well as on Projects in the Visual Arts field, in archaeology, history 
and other Culture Activities. Working Groups and new Projects are promoted. Scientific and technical 
demonstrations are presented. Technology and Art Exhibitions are promoted. 
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% European Commission Projects and Plans regarding Cultural Heritage 

% Mediterranean Initiatives in Technology for Cultural Heritage: Synergy with European & 
International Programmes 

“* 2D -—3D Digital Image Acquisition 

% Leading Edge Applications: Galleries, Libraries, Archaeological Sites, Museums & Historical Tours 
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«* Management of Museums by using ICT Technology: Documentation, Access, Guides & Other 
Services 
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9,30 — 13,00 WORKSHOP p. 11 
14,15 — 15,30 OPENING OF CONFERENCE p. 12 
15,45 — 17,25 SESSION 1 p. 12 
17,25 — 18,40 SESSION 2 p. 13 
10,00 — 13,00 TECHNICAL EXHIBITION p. 18 
19,00 — 21,30 SPECIAL EVENT p. 18 


Thursday, 9 May 


9,30 — 11,45 INTERNATIONAL FORUM ON p. 15 
“CULTURE & TECHNOLOGY” 
12,00 — 13,15 SESSION 3 p. 16 
14,50 — 16,05 SESSION 4 p. 16 
16,20 — 17,35 SESSION 5 p. 17 
10,30 — 13,00 ART EXHIBITION p. 19 
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WORKSHOP 


ROOM A 

WORKSHOP INNOVATION AND ENTERPRISE — INNOVAZIONE E IMPRESA 
(Italian Language) 

9,30 — 13,00 

Chairman: Enrico Bocci, Vice-Presidente Confindustria Firenze, Firenze 


Technological requirements in the Cultural Heritage field are outlined and opportunities for 
Italian Enterprises and SME’s working in the field, using new technologies, are presented. 
Regional and National Applied Research Programs in Italy are described. 

Activities by National Organizations and Firms working in the area of Telecommunications, 
Informatics, Environment and Infomobility are presented. 

Funding by European Commission is considered, with particular reference to multimedia 
and telematics for Cultural Heritage. Special consideration is given to the EC Plan 
HORIZON 2020 and the following ones. 

Initiatives regarding the “know-how” transfer from Research Organizations to the Industrial 


Sector are described, in particular to create Start-Ups and new Enterprises. 


Invited Speakers: - Andrea Arnone, Pro-Rettore al Trasferimento Tecnologico e 
Presidente di CsaVRI, Università degli Studi di Firenze, Firenze 
- Laura Castellani, Responsabile del Settore Infrastrutture e 
Tecnologie per lo Sviluppo della Società dell’Informazione, 
Regione Toscana 
- Paola Castellacci, VAR GROUP 


Speakers include: - Stefano Cinquini and Luigi Carfagnini, TELECOM ITALIA 

- Renzo Zampini, INFOCAMERE 

- Andrea Calistri, SAPAF Srl, Firenze 

- Claudio Tasselli, Qu.In. Srl, Calenzano, Firenze 

- Riccardo Bruschi and Luca Bencini, T.T. TECNOSISTEM Spa, Prato 

- Carlo Lastrucci and Luca Lastrucci, Powersoft Spa, Firenze 

- Gianluca Vannuccini, Servizio Sviluppo Infrastrutture Tecnologiche, 
Comune di Firenze, Firenze 

- Giacomo Bucci, Università degli Studi di Firenze, Firenze 

- Francesco Mati, Piante MATI, Pistoia 

- Giuliano Benelli, Università degli Studi di Siena, Siena 

- Franco Guidi, NEUMUS Srl, Firenze 
and Roberto Piermarini, ICT Consulting Expert 

- Paola Imposimato, Studio Creazioni di Design e Arti Grafiche 
e Pittoriche, Firenze 


Closing: - Cecilia Del Re, Assessore Sviluppo Economico e Turismo, 
Comune di Firenze, Firenze 


13,00 Lunch Break 


11 


ROOM A 


CONFERENCE 


Wednesday, 8 May 


Chairmen: 


Vito Cappellini, University of Florence 


Enrico Del Re, University of Florence 


14,15 Opening: 


Eugenio Giani, 


President of Consiglio Regionale della Toscana 


Enrico Vicario, 


Director of Dipartimento di Ingegneria dell’ Informazione 
Universita di Firenze 


Gabriele Gori, 


General Director of FONDAZIONE CR FIRENZE 


Paolo Castellacci, 


President of GRUPPO SESA 


15,30 Coffee Break 

ROOM A 

15,45 SESSION 1 — STRATEGIC ISSUES 
Chairman: 


"The DAFNE Project: Human 
and Machine Involvement” 


“Restored Paintings and Visual Perception: A 
Proposed Protocol to Study Emotional and 
Cognitive Involvement in Art” 


Paolo Blasi, University of Florence, Florence, Italy 


V. Cantoni!, L. Lombardi’, 

G. Mastrotisi?, A. Setti! 

‘Dept. Electrical, Computer and Biomedical 
Engineering, University of Pavia, 

Pavia, Italy 

*Novaria Restauri Srl, 

Novara, Italy 


V.A. Sironi!, A. Banzi?, R. Folgieri”5 

!Centre on the History of Biomedical Thought, 
University of Milano Bicocca, and Cespeb 
Neuroaestetic Laboratory, 

Milano, Italy 

?Cespeb Neuroaestetic Laboratory, 

Milano, Italy 

3Department of Philosophy "Piero Martinetti", 
Universita degli Studi di Milano, 

Milano, Italy 
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“Technology and the Art Market: Access, 
Protection, and Transparency” 


“Evolutionary Dynamics and Computational 
Aesthetics: Experiments in Minimalist Blends” 


“Electrical Networks as Media: Post - Humanist 
Media Archaeological Analysis of the 
Digitalization of Electric Networks” 


ROOM A 


Frances Oglesby 
Artive Inc., 
Atlanta, GA, U.S.A. 


Stefano Kalonaris 

RIKEN Center for Advanced Intelligence Project 
(AIP), 

Tokyo, Japan 


Mika Laakkonen, Ville Kivivirta 
Faculty of Social Science, 
University of Lapland, 
Rovaniemi, Finland 


17,25 SESSION 2 - NEW SCIENCE AND CULTURE DEVELOPMENTS & 


APPLICATIONS 


Chairman: Edoardo Calia, FONDAZIONE LINKS, Turin, Italy 


“It’s Time to Give Back” (03 LAB)” 


“A Blockchain-Based Support to 
Safeguarding the Cultural Heritage” 


“Educate to Wellness by the 
Emotion of Light” 


Massimiliano Zanoni’, Jean Paul Carradori?, 
Qing Li? 

103 Lab, 

Milan, Italy 

2Shanghai International Interior Design Festival 
03 Lab 

3art Projects 03 Lab, 

Beijing, China 


Mariano Basile!, Gianluca Dini‘, Andrea 
Marchetti’, Clara Bacciu’, Angelica Lo Duca? 
*Dept.of Ingegneria dell’Informazione, 
University of Pisa, 

Pisa, Italy 

"Institute of Informatics and Telematics, 
National Research Council, 

Pisa, Italy 


G. Alfarano’, A. Spennato? 

‘Department of Architecture - Design Campus, 
Laboratory Design Model and Laboratory Smart 
Lighting Design, 

University of Florence, 

Florence, Italy 

*Department of Architecture - Design Campus, 
Laboratory Design Model, 

University of Florence, 

Florence, Italy 
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“Robot Surgery for Total Knee Arthroplasty” Michele D’Arienzo* 
Lawrence Camarda’, 
Antonio D'Arienzo’, 
*Department of Orthopaedic Surgery, 
University of Palermo, 
Palermo, Italy 
*Department of Orthopaedic Surgery, 
University of Pisa, 
Pisa, Italy 
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Thursday, 9 May 
ROOM A 


9,30 INTERNATIONAL FORUM ON “CULTURE & TECHNOLOGY 
Chairman: Vito Cappellini, University of Florence, Florence, Italy 


The structure of the FORUM is presented. 
Actual developments and perspectives are outlined, regarding Culture and Technology. 


- Cooperation Groups 

- Proposed Projects 

- Funding Opportunities 

- European Commission Plans 


Opening: - Eugenio Giani, President of Consiglio 
Regionale della Toscana 


Speakers Include: 
- Cristina Acidini, President Accademia delle 
Arti del Disegno, Florence, Italy 
- Edoardo Calia, Deputy Director, 
FONDAZIONE LINKS, Turin, Italy 
- Alberto Del Bimbo, Centro per la 
Comunicazione e l'Integrazione dei Media, 
University of Florence, Florence, Italy 
- Luigi Rucher, Technical Director THALES 
ITALIA, Italy 
- David Feldman, Vice President 
THE MONA LISA FOUNDATION, Zurich, 
Switzerland 
- Monica Carfagni, Full Professor of Industrial 
Engineering, University of Florence, Florence, 
Italy 
- Carlo Francini, Florence Municipality, 
Florence, Italy 
- Paolo Zampini, Director of Conservatorio di 
Musica Luigi Cherubini, Florence, Italy 
- Francesco Bellini, University of Rome “La 
Sapienza” and Research Director of 
EUROKLEIS, Rome, Italy 
- Giovanni Gasbarrone, Innovation Advisor, 
Rome, Italy 


11,45 Coffee Break 
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12,00 
Chairman: 


“NEMOSINE: Innovative packaging solutions for 
storage and conservation of 20th century 
Cultural Heritage of artefacts based on 
cellulose derivate” 


“Living in the Present: Using 3D Mixed Reality 
Technology to Enhance Guest Experiences 
with Museum Objects and Collections” 


“Combining RTI & SFM 
A Multi-Faceted Approach to Inscription 
Analysis” 


“Chateau de Chambord, a Perfect Order” 


SESSION 3 — NEW TECHNICAL DEVELOPMENTS & APPLICATIONS 
Andrea De Polo Saibanti, Fratelli Alinari IDEA Spa, Florence, Italy 


Andrea De Polo Saibanti 
Fratelli Alinari IDEA Spa, 
Florence, Italy 


D. Marshall’, R. Hite’, J. Hoffman! 
‘Museum of Texas Tech University, 

Texas Tech University, 

Lubbock, Texas, U.S.A. 

*Department of Curriculum & Instruction, 
Texas Tech University, 

Lubbock, Texas, U.S.A. 


Moshe Caine’, Michael Maggen’, 
Doron Altaratz’, 

Hadassah Academic College, 
Jerusalem, Israel 

"Israel Museum, 

Jerusalem, Israel 


Jeanette Zwingenberger’, Bertrand Triboulot? 
1Université Paris 1 Panthéon-Sorbonne, 

Paris, France 
* Direction régionale des affaires culturelles d'Île-de- 
France, Service régional de l'archéologie, 
Paris, France 


13,15 Lunch Break 

ROOM A 

14,50 SESSION 4 - CULTURAL ACTIVITIES — REAL AND VIRTUAL 
GALLERIES AND RELATED INITIATIVES 

Chairman: 


Paris, France 


“Architectonic design for mediating 
Cultural Heritage ” 


“Innovative methodologies and technologies. 
Civil liability and "genetic code" of a work 
of art” 


Jeanette Zwingenberger, Université Paris 1 Panthéon-Sorbonne, 


Dominik Lengyel’, Catherine Toulouse” 

BTU University of Technology Cottbus-Senftenberg, 
Cottbus, Germany 

*Lengyel Toulouse Architects, 

Berlin, Germany 


Sara Penco 
Restorer and creator of Smarticon Project, 
Rome, Italy 
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“Information Technology in Conservation and 
Restoration of Art Works: Perspectives of 
Ukrainian Universities Project” 


“Museum Collections Using Information 
and Digital 3D Technologies” 


16,05 Coffee Break 


ROOM A 


16,20 
Chairman: 


“Contemporary Methods of Functional 
Harmony Teaching in a High School Context” 


“The Aesthetics of Volumetric Photography for 
Virtual Reality” 


“Optimising 3D cultural environments with 
large amount of texts for 3D Web. The Santo 
Stefano Lapidarium to the dead soldiers of the 
Great War, a case study” 


"A techno social collaborative platform to 
optimize Cultural Heritage funding: final 
results from the validation phase" 


Roman Mykolaichuk’, Tetiana Tymchenko?, 
Alisa Mykolaichuk?, 

Iryna Somyk-Ponomarenko?, Vira Mykolaichuk?, 
Antonina Mykolaichuk? 

‘Taras Shevchenko National University of Kyiv, 
Kyiv, Ukraine 

*The National Academy of Fine Arts and 
Architecture, 

Kyiv, Ukraine 

3State University of Telecommunications, 

Kyiv, Ukraine 


Tinatin Mshvidobadze, Sopio Mshvidobadze 
Gori State University, 
Georgia 


SESSION 5 — ACCESS TO THE CULTURE INFORMATION 
James Hemsley, EVA Conferences International, U.K . 


Anna Shvets 

Institute of Cultural Studies, 

Maria Curie-Sklodowska University in Lublin, 
Lublin, Poland 


Daniel Buzzo 
University of the West of England, 
Bristol, UK 


Beatrice Chiavarini, Daniele De Luca, Antonella 
Guidazzoli, Maria Chiara Liguori, 

Silvano Imboden, Luigi Verri 

VisitLab Cineca, 

Bologna, Italy 


F. Spadoni!, F. Tariffi?, R. Rossi, S. Lusso’, 
‘Rigel Engineering Srl, 

Livorno, Italy 

“Space Spa, 

Prato, Italy 
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SPECIAL EVENT 


Wednesday, 8 May: 19,00 - 21,30 


Visit to ACCADEMIA DELLE ARTI DEL DISEGNO 
President Cristina Acidini 


with Concert by Paolo Zampini, Director 
of CONSERVATORIO DI MUSICA “LUIGI CHERUBINI” 
of Florence 


in Cooperation with ANTICA COMPAGNIA DEL PAIOLO 
President Anna Bini 


and 


“Wellcome Drink” 


EXHIBITION 
ROOM B 


Wednesday 8 May: 10,00 - 13,00 


TECHNICAL EXHIBITION 


Coordinator: Marco Cappellini, CENTRICA Srl 


Advanced Technologies will be presented, with Technical Demonstrations, 
regarding in particular: 


- Augmented Virtual Reality Immersive Systems 
- 3D Digital Models Protection 
- Uffizi Touch® Cloud Edu for Schools 


by some Enterprises, Leaders in the above Areas. 


Thursday 9 May: 10,30 — 13,00 


ART EXHIBITION 
Coordinator: Vito Cappellini, EVA 2019 FLORENCE Organizer 


Digital Presentations of Museums, Art Institutions, Virtual Galleries and Art- 


Works will be performed. 
In particular Digital Art-Works of LEONARDO DA VINCI will be presented. 


in Cooperation with ANTICA COMPAGNIA DEL PAIOLO, 
President Anna Bini 


Events Organization: INN-3D Srl, Empoli (FI) 


Official Television TVL Spa Pistoia 


NEXT EVA EVENT 


8-11 July 2019 EVA 2019 London 


Website: http://www.eva-london.org/eva-london-2019/ 
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THE DAFNE PROJECT: HUMAN 
AND MACHINE INVOLVEMENT 


V. Cantoni*, L. Lombardi*, G. Mastrotisi**, A. Setti* 
"Dept. Electrical, Computer and Biomedical Engineering, University of Pavia, Italy 
**Novaria Restauri s.r.1, Novara, Italy 
virginio.cantoni@unipv.it, luca.lombardi@unipv.it, 
giovanna.mastrotisi@rotarycultura.it, alessandra.setti@unipv.it 


Abstract — The DAFNE challenge originated with the aim of involving autistic subjects in 
the post-seismic re-composition of destroyed frescoes, trying to develop their social 
involvement while favoring their inclusion in productive activities, enhancing their 
peculiarities and abilities, and promoting their potential. Our goal became to produce also a 
software tool that applies the most advanced computer techniques, such as Machine 
Learning and Deep Learning. As different tracks are proposed, this initiative can bring to 
different applications. It widened then in a research of up-to-date artificial intelligence 
solutions suitable to the restoration of collapsed buildings, promoting their re-use, 
enhancing cultural heritage conservation. 


INTRODUCTION 


We focus on the digital anastylosis of frescoes. The reconstruction of an artwork destroyed 
by natural events, such as earthquakes, or wars, is a very complex task. Several attempts to 
restore collected fragments, based on traditional methods, have been done in the last 60 years, 
but without great success. Since 1998 [1] many efforts have been devoted to the restoration of 
the inestimable frescoes by Andrea Mantegna — located in the Ovetari Chapel of the Eremitani 
Church, in Padua (Italy), and destroyed in a bombing during the Second World War — by using 
mathematical methods and computer techniques based on comparison with an old gray level 
image of the fresco prior to the damage. Unfortunately, with this technique only a partial 
reconstruction of the fresco was possible: from the 78.561 pieces, mostly with an area of 5-6 
cm’, less than 8% of the artwork was covered. 

A variety of different approaches have been proposed in the last decade. A first broad 
taxonomy of these techniques [2] is based on the main matching factors: color information [3], 
geometric outline characteristics [4] or both together (shape as well as sample colors along the 
border of the pieces) [5, 6, 7]. 

A number of proposed methods has been explicitly based on the optimization of specific 
remapping problems. An important one has been defined ‘colorization’; as an example, in the 
quoted case of the Ovetari Chapel, the simple 8% of coverage of the frescoes was certainly 
instrumental to a high fidelity to the fresco re-colorization, which is a crucial issue in art 
restoration [8, 9]. A second example refers to image ‘completion’, focusing on the cases of 
occlusion areas that produce regions to be completed [10, 11]. A peculiar case is the one of 
non overlapping images, in which images are extrapolated to cover the gap between them 
through alignment and inpainting [12]. A related problem is image ‘melting’, in cases of 
transition regions in which chromatic and structural properties change gradually so that a patch 
transform allow the implementation of image interpolation and morphing by combining 
inconsistent images [13, 14]. 

It is worth to mention another approach, based on the analysis of fracture patterns in order 
to model the cracking process so that it will be possible to predict the patterns and help future 
computer assisted anastylosis [15]. 

In the very last years, two very promising innovative solutions have been proposed, both 
computation intensive, but now supported by the high performances of the new technology. 
The first one exploiting a genetic algorithm approach [16] and a second one, becoming more 
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and more popular, exploiting machine learning and deep learning [17]. In this connection, we 
are proposing this challenge and we foreseen too far, may mature in the next couple of years, a 
‘blind’ anastylosis in which no data on frescoes content and semantics are known [18]! 


THE DIGITAL ANASTYLOSIS OF FRESCOES CHALLENGE (DAFNE) 


The challenge’s goal is to collect the best solutions to virtually recomposing destroyed 
frescoes, starting from the digitalization of their broken collected elements. This is not meant 
to replace the restorer’s or archaeologist’s skills and knowledge, but to be an additional tool in 
the restoration task. In this context, it can be interpreted as a very challenging ‘puzzle’ and 
finding the global position and rotation for each piece, so that all fragments jointly reconstruct 
the original surface, is the final objective. Critical issues are due to: i) the number of randomly 
mixed fragments is usually huge; 11) fragments are mostly corrupted, and with general irregular 
shapes; iii) mismatch of the boundaries of the collected eroded pieces; iv) some pieces have 
gone irretrievably lost; v) due to extreme fragmentation, presence of spurious/distractors 
elements, as pieces from different frescoes could be involved in the building collapse. 

The challenge entitles participants to use, at their discretion, a number of different cases of 
destroyed well-known frescoes that have been simulated in order to populate a dataset 
containing fragmented pieces, useful for the development and testing phases of the challenge. 
Five different parameters have been used for the generation of elements, in order to respond to 
the critical issues previously listed: A, the number of fragments; B, the type of random 
distribution; C, the percentage of missing parts; D, the percentage of spurious fragments; and 
E, the ratio between the fragment area after the erosion and the original area in the fresco plane 
tessellation. 

Participants are invited to develop solutions that re-assemble a given set of pieces, 
discarding spurious ones, and are required to submit the reconstructed image generated by 
their application — showing the fragments properly placed — which will be evaluated in 
accordance with quality requirements and metrics. 

Methods developed for DAFNE can be tested on cases for which ground truth fragments 
alignments are available for verification. ANASTYLOSIS DBI collects for each fresco image 
more sets of digital fragments, mixed with spurious elements (distractors), that participants 
have to re-assemble. Two phases will be considered: 1) a development phase, during which 
registered participants will be provided access to DB1, with fragments and solution (an image 
file with each fragment provided in the right position and a corresponding table with location 
and rotation for each fragment); 2) a successive testing phase, during which registered 
participants will be provided with a dataset DB2 of fragments from the image of five frescoes 
—with distractors from other frescoes — and the original image of the corresponding fresco, but 
no ground truth solution will be provided. 


FRAGMENT GENERATION: THE CRACKING PROCESS 


Studies have been done on the topic of “natural” fragmentation, to model a method for 
generating surface crack patterns, and different solutions have been proposed. For example, 
the Berkeley Surface Cracking Toolkit [19] model a hierarchical fracture process that appears 
in materials such as mud, ceramic glaze, and glass, that the user can control using a set of 
simple parameters for the characteristics and appearance of the cracks; others [16] run their 
experiments with data from Princeton Synthetic Fresco, a plaster wall painting created and 
fractured by Greek archaeologists for the purpose of testing reconstruction algorithms. Some 
authors [2] used data from artifact/fresco broken based on a fragmentation of dry mud; other 
[20] evaluated their approach using scanned frescoes from archaeological excavations (but 
with a dataset limited to 1200 fragments), as well as a modern-day “synthetic” fresco data set, 
created and then broken into pieces to have fragments similar to the ones found at 
archaeological sites. In addition, other authors [20] present a system for acquiring images, 
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geometry, normals, and associated metadata, for small objects such as fragments of wall 
paintings. 

Starting from examples of real fragments of destroyed frescoes we based our simulated 
fragments generation on some generic considerations: the fragments must be in different sizes, 
they do not have sharp edges, they may be convex or concave, we have to discard the too 
small fragments (so a threshold has to be fixed in order to eliminate the fragments that result 
too small after the fragmentation), size and shape of the fragments have not to be dependent 
from their position inside the image. 

An application tool has been developed to generate some examples of plausible 
fragmentation, to populate the challenge dataset. Figure 1 shows the fragmentation obtained 
with two different types of distribution: concentrated and uniform. Figure 2 and Figure 3 show 
two examples of plane tessellations and corresponding ground truth cases. 
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Figure 1. Above: Plane tessellation with a uniform distribution adopting the Manhattan 
distance (detail of the fresco “The Last Judgment” by Michelangelo Buonarroti, Sistine 


Chapel, Vatican City, 1536-1541). Below: Plane tessellation with a concentrated distribution 
adopting the Euclidean distance. 
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Figure 2. Above: Plane tessellation with parameters: A=750; B=random distribution generated 
adopting the Manhattan distance; C=7.5%; D=7.5%; E=0.10. Below: Ground Truth, with 
original image in background and overlapping fragments (image of the fresco “The Plague of 
Milan” by Cesare Nebbia, Collegio Borromeo, Pavia — 1604). 
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Figure 3. Above: Above: Plane tessellation with parameters: A=253; B=random distribution 
concentrated distribution adopting the Euclidean distance (annular measures p1=3px, p2=10px, 
t=0.5); C=7.5%; D=7.5%; E=0.10. Below: Ground Truth, with original gray image in 
background and overlapping color fragments (detail of the fresco “The Last Judgment” by 
Michelangelo Buonarroti, Sistine Chapel, Vatican City,1536-1541). 


A graphical interface (Fig. 4) has been designed for handling the fragments to be 
reassembled. This has been studied and will soon be tested by people belonging to the autism 
spectrum. This tool allows them to move and rotate each single piece in the position that they 
suppose it is right. Automatically a table identifying each moved fragment with its final 
location and rotation will be listed for restorers’ usage during the physical phase of artwork 
reassembling. 
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Figure 4. The multi-platform application which provides a set of tools to reconstruct images 
from pieces, like a puzzle. The interface has been designed to let the user move and rotate the 
fragments, store them on the provided panel and save the progress, in order to retrieve their 
position for further usage. The lower part of the screen shows a list of fragments to be 
reassembled, while the majority of the display is occupied by the image that has to be 
reconstructed (in figure, a detail of the fresco “Herod’s Banquet” by Fra Filippo Lippi, located 
in Prato Cathedral, Italy). Both mouse and keyboard bindings are provided in order to assure a 
comfortable usage and allow fast and accurate operations. The fragment with blue borders is 
the active one. 


CONCLUSIONS 


This initiative can foster different strategies. The topic is of interest for the Pattern 
Recognition community and advanced computer techniques such as Machine Learning and 
Deep Learning can be applied to improve methods and applications. 

Restoration of damaged frescoes is a crucial task in cultural heritage preservation and these 
kind of solutions can result in a precious additional tool for restorers. Furthermore, the 
involvement of autistic subjects in the post-seismic recomposition of destroyed frescoes, can 
be both an inclusion opportunity as well as an enhancement of their peculiarities and abilities. 
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Abstract — In this ongoing study we are interesting in investigating visual skills, cognitive 
understanding of an artworks and the observers’ brain response to the aestetic perception of the 
artwork. The study will involve a wide number of participants to the experiment, making our 
investigation a unique occasion to confirm data and outcomes from previous works described in 
literature. 


INTRODUCTION: AIMS AND MOTIVATION 


For some year neurosciences have begun to take an interest in art to try to understand the 
neurobiological bases of artistic production and to know which neurophysiological networks allow 
us to grasp how "beautiful" and/or "pleasant" or "unpleasant" and/or "ugly" a painting, a sculpture 
or an architectural work is. 

Every time an aesthetic judgment is formulated, different areas of our brain are activated. The 
techniques of recording cerebral electrical activity (EEG, Electroencephalography) and the modern 
neuroimaging techniques (as Functional Cerebral Magnetic Resonance) allow to investigate what 
happens in the brain of people who observe an artistic work. If a "pleasing" work enters our field of 
vision, for which we formulate a positive aesthetic judgment, the medial orbital-frontal area is 
activated, together with the occipital cerebral areas designated for vision [10, 9]. If, on the other 
hand, our aesthetic judgment is negative, the left motor cortex is activated [8]. Finally, if we remain 
indifferent, the lower part of the cingulum and the parietal cortex enter into action [16]. Different 
areas are also affected when the subject of the artistic work changes: landscapes, portraits and faces, 
still lives or objects activate different areas, as if each type of representation corresponded to a 
different cerebral "micro-conscience"[15]. 

In this field, the neurology of art or neuroaesthetic, as named by Semir Zeki [20, 12], represents 
an approach that considers the artistic-aesthetic analysis as a function of the neurological impact 
and the neurophysiological study of the work of art. Many information has now been acquired in 
this area, but often these studies investigate only some functional and neurophysiological 
parameters and are conducted on a limited number of people. 

In this context, our study, developed by CESPEB (Centro studi sulla Storia del Pensiero 
Biomedico) of the Universita di Milano-Bicocca with the involvement of his Neuroaestetic 
Laboratory, born from interdisciplinary collaboration between Department of Phylosophy of the 
Universita degli Studi di Milano and Department of Medicine and Surgery of the Universita degli 
Studi di Milano-Bicocca, has a double aim: as first, to confirm the outcomes from previous 
researches through a significantly wider data sample; then, we also aim to add a series of 
investigations usually not used in research. Indeed, the project includes a first phase during which 
the already known data reported in the literature as related to the vision of pictorial works using 
non-invasive easy-to-use tools will be taken back to obtain more reliable quali-quantitative 
indicators. 
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Sometimes visitors cannot look at and understand a restored artistic exhibit displayed in a 
museum. Looking is a learned skill [17, 18, 19, 2, 21] that is neither innate nor spontaneous. In this 
paper, as previously mentioned, we introduce a promising approach consisting of analysing 
subjects’ brain signals collected by an EEG-based device. The approach is currently in use in our in 
fieri project concerning museums and art exhibition in Lombardy Region. 

In next paragraphs we will introduce the protocol in use and a discussion about our choices, in 
the light of the outcomes we expect from the study. 


BACKGROUND 


In our study we are interesting in investigationg visual skills, cognitive understanding of an 
artworks and the observers’ brain response to the aestetic perception of the artwork. 

To collect data useful to analyse these aspects we chose to use some devices and methods, better 
described in next paragraph about the adopted experimental protocol. Specifically, we chose to use 
Brain Computer Interfaces (BCIs) and an eye tracker to analyze brain data and visual skills related 
to the emotional and aestetic impact of an artwork on observers and the priming methodology to 
investigate their cognitive response. We also aim at using data coming from the measurement of the 
skin conductance, collected while individuals are looking at an artwork. 

As visual literacy experts state [3, 4, 13, 14], visual skills are not to be confused with vision, 
colour vision, disease, and various anomalies. Visual perceptual motor skills and ocular motor skills 
are the main visual skills categories. These skills are developed after birth. Visual perceptual motor 
skills process visual information and affect eye/ body movements. They encompass abilities such as 
visual memory, visual-spatial skills (e.g., mapping locations), visual analysis (e.g., discrimination), 
visual-motor coordination, visual-auditory integration (e.g., matching sound and image), and 
visualization. Ocular motor skills involve eye movement control and focus control. 

A work of art is composed of different layers of meanings including also the restored version of 
the object. How can we understand these meanings? 

According to Angela Lawler and Susan Wood [11] there are five steps which teach people how 
to look at art. We have adapted this methodology to the observation of restored artworks. 

The first step is to observe artworks in silence, secondly time it is taken to describe the artwork 
objectively. The third step analyses contents, such as colour, balance, space, line, value, and 
technique. The fourth step tries to interpret the works of art exploiting what museum-goers know 
and have seen. Finally, museum-goers try to make a critical judgement of the artwork. Judging art 
requires fair and logical consideration. Angela Lawler and Susan Wood recommend taking time, 
because reading art is a slow, thoughtful, and exciting process of discovery. 

There is a final step we would like to add in order to foster visual skills: comparing the artwork 
before and after the restoration, asking the visitors to list the diversities between both pictures (the 
restored one and a picture of the artwork before the restoration). 


THE PROTOCOL 


The ability to analyse the formal qualities of a restored artwork is an important step of the art- 
making process understanding, depending on several factors and involving visual skills, the brain 
response to stimuli [5, 6, 7] and the body related reaction to such perceptions. Here is a streamlined 
example of a tentative method of collecting artwork-related visual skills data through the collection 
of different brain and body response. 
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The approach 


We decided to adopt an ecological approach, that is we decided not to isolated the individuals 
from the environment. Specifically, we chose to collect data while individuals submitted to the 
experiment are visiting a Art exhibition so that we could collect real response to stimuli from 
artworks in a real environment, also influenced by distracting factors, noises, fatigue due to the tour. 
Indeed, we intend to analyze a real scenario, that is the condition in which individuals usually look 
at artworks. 

The data will be collected on a selected number of paintings, specifically ten, and in conditions 
of spontaneous and "guided" vision, allowing us to elaborate fundamental information to 
understand the neurobiological mechanisms involved in the neuroaesthetic perception. 


Participants to the experiments 


A population of several hundred people will be studied, namely, a large number of visitors to 
temporary exhibitions and permanent art museums in Lombardy Region. The sample will be 
composed by individuals of different nationality, sex, age and education level, so that we could 
have a large set of data to perform our analysis and comparisons. 


Materials 


During the vision of the paintings the skin resistance will be recorded to know the 
neurovegetative involvement of the subject, the eye movements with an eye-tracker, which 
highlights the parts of the work most observed by the visitor, and the brain electric activity with an 
EEG-based Brain Computer Interface (BCI) headset [1] to observe the changes of the electric 
activity in brain areas solicited and activated by the vision of the masterpiece. 

Because of the ecological approach we chose to use easy-to-wear devices, such as: 

- the Pupil headset for the eyetracking (https://pupil-labs.com/store/); 

- the Mindplay (http://mindplay.com/) EEG-based BCI; 

- tools developed using the bitalino (https://bitalino.com/en/) to measure the skin conductance 
and possible other biofeedbacks. 

All these devices are used during the performed experiment to assess and collect data on the 
museum-goers' visual skills and emotional/cognitive reactions to the artworks. The listed devices 
have been chosen because they are low-cost and portable. Also, they can be connected via bluethoot 
or WI-FI to a computer, so the visitors will be free in movement and not under the effect of anxiety 
often induces by more complex devices. Moreover, they are perfectly comparable to the 
performance of medical devices. Indeed, for this latter reaons, they are widely used in research. 

Thanks to devices listed above, the participants will visit the museum without any restrictions. 

Individuals will be informed about the experiment and provided by a written consent to take part 
in the experiment. 

Also, a questionnaire will be prepared, with the aim to verify what they experienced and what 
they remember after the visit. 


Procedure 


At the onset, participants wear the BCI devices and read the instructions which they will then 
paraphrase back to the experimenter. Participants are asked whether they have any doubts as to 
what they have to do. 

The museum-goers freely observe the restored artwork for 1 minute. Subsequently the 
experimenter asks each participant separately to describe the artwork without any specific 
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instruction. After that he guides the analyses of the visual features of the restored artwork following 
a preset list of visual contents such as colour, balance, space, line, value, and technique. 

The next step is very delicate since it requires an interpretation of the artwork made by the 
participant. After collecting the participant's interpretation, the experimenter explains the 
discoveries made during the restoration in order to give a correct point of view on the basis of 
scientific results. The museum-goer is invited to try to express a critical judgement of the artwork. 

To facilitate the complete understanding of the artwork, the participant is encouraged to look at a 
copy of the artwork before the restoration and to list the differences between the two pictures (the 
restored one and a high-resolution picture of the artwork before the restoration). 

After the museum exhibition tour, participants, still wearing Mindwave in order to collect more 
EEG signals, complete a questionnaire about the visual features of the corresponding restored 
artworks. Indeed, at the end of their tour, the participants will be asked to answer some questions 
about the masterpieces chosen for the experiment in order to check what they remember. This latter 
step will allow us to analyse also the effect of the priming approach on the visitors’ cognitive 
process. 


DISCUSSION AND CONCLUSION 


The novelty of the study consists in many points. First of all, a population of several hundred 
people will be studied, making the project one of the larger experiment ever performed in the field. 
Also, the ecological approach is very interesting, allowing us to collect data on a selected number of 
paintings in conditions of spontaneous and "guided" vision, to elaborate fundamental information to 
understand the neurobiological mechanisms involved in neuroaesthetic perception in a real 
enviroment. This approach will allow us to obtain important information giving us practical and 
useful guidelines for exhibitions and museums. Another aspect is related to the new cognitive 
technologies adopted (EEG-Based BCI headset) and Artificial Intelligence approaches (eye 
tracking, facial expression recognition) combined with physiological measurements (skin 
conductance, possible ECG, EMG) allowing freedom of movement to the individuals participating 
to the experimental sessions. 
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ABSTRACT - The increasing development of digital networks and technologies is changing the 
way that data is collected, digitized, managed, and searched. These strides not only provide 
platforms for tangible materials to be digitally recorded and exchanged, but also raise the standard 
of accessibility, methods of protection, and ease of research. Can the current climate of 
technological advancements assist in making provenance research a priority for institutions, 
companies and individuals involved in the art market? 


This paper will examine what potential impact future technological opportunities and 
interpretations of ‘Big Data’ could have on cultural property protection, provenance research, and 
due diligence processes. 


INTRODUCTION 


The longstanding identity of an art world rooted in tradition and noted for resistance to 
change is being challenged by the increasing availability of digital technologies within business 
and research practices. Today's media teems with buzzwords such as ‘blockchain’, ‘Artificial 
Intelligence’, and ‘Big Data’ thus pressuring industries across the board to consider how they 
will progress in this time of technological renaissance. How are these trends targeting or 
inspiring thought-leaders in the art world? As our globe becomes more connected through 
endless channels of communication and networks for exchanging information, does this create a 
demand for quicker, easier, and cheaper modes of research and transactions? And if so, how 
might these impact the standard of business practices in the art trade? 


To answer these questions, throughout this paper we will explore where we are seeing 
technologies applied to art practices, analyze what this means for the future standard of art 
business, and point out the areas of hope and resistance present throughout the industry in 
reaction to the promises made by forthcoming technologies. 


1. Technology and the Art Market 


To begin, let us establish a basic understanding of the current state of technological 
advancements and what has informed their development. Less than thirty years ago the public 
introduction of the world-wide web greatly stimulated the concept of digitally recording, storing, 
and exchanging information. It was a nuanced idea that tangible materials could be held, 
accessed, and traded without geographical limitations. From there, more complex conversations 
about data collection began and continues to be explored, informing a significant increase of 
electronic research tools and data collection platforms. This brings us to the buzzwords 
mentioned above. 
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Defined in their most basic form, ‘blockchain’; digital ledgers, ‘Artificial Intelligence’; machines 
mimicking cognitive functions, and ‘Big Data’; extensively large amounts of data, are all tools 
intended to set high standards for how data should be handled and to build trust among users. 
These technologies of the future are for “those hoping for a utopian future of information sharing 
and the end of big business dominating the use of personal data”. While some research may 
suggest that aspects of these technologies are “more a narrative than any specific technology” it 
is plain to see that they have “sparked the imaginations of people across sectors to consider how 
digital technologies” could alter the landscape of business and research methods in a very 
positive way [1]. As these technologies are developed and implemented into business practices, 
the endless possibilities of collecting and sharing information leads to a demand for data 
democratization. Can democratization of this nature find a place in the notoriously private, 
opaque world of art? 


This question brings us to the second establishment that needs to be made: let’s define the 
fundamental qualities of the art market. Plainly stated, the art sector is discreet and highly 
consolidated. Art sales continue to be largely private in nature, with physical transactions 
dominating the market and the “lion’s share of profits” going to “a fairly small number of top 
galleries and auction houses” [1]. While sectors such as banking, insurance, and real estate are 
highly regulated markets governed by well-established protocols to protect their participants, the 
art market remains opaque and vulnerable to tampering. We see this in the way that the art 
market is “rife with illicit business. All too often owners of art works, even, or especially, at the 
higher end, use assets to launder money, or at least as a value store that is beyond the scope of 
regulatory bodies.” Additionally, such works are often traded completely discreetly and without 
tax. Art is frequently stored by its owners “in offshore locations, or jurisdictions which are 
beyond the reach of regulators”. For such individuals, the lack of transparency in the art market 
is of critical importance. Accordingly, the companies which service this high-end clientele also 
greatly value the discreet and opaque nature of the market. Therefore, there are more vested 
interests to keep the art market opaque and stop it from adopting” technologies that would bring 
transparency, shedding light onto such discrepancies [1]. 


So, it is evidenced that the opaque nature of the market is directly connected to its problematic 
qualities. This is why perhaps even more necessary than other industries, trust is absolutely 
essential for the preservation and success of the art world [1]. Fearful of the illicit activities 
mentioned above, artists, buyers, and sellers seek access to information regarding the pieces they 
are trading as well as who they are trading with. Advancements in technologies and the promises 
they are making of transparency and increased ease of access to information is instilling great 
hope for the industry. 


2. Digital Advancements, Research, and Accessibility 


Over the past decade we have seen an undoubtedly dramatized flair cast upon the topics 
of looted, stolen, missing, and forged artworks. Increased media attention through films such as 
The Monuments Men (2014) and Woman in Gold (2015) have gathered a wide audience of 
intrigued viewers, entranced by the mystery and suspense of illicit art activities. While these 
stories, based in truth, focus on the catastrophes of World War II looted art that have ongoing 
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repercussions, the same threats continue to retain relevance in the contemporary marketplace. As 
plainly stated in a 2018 report by the University of Oxford and Alan Turing Institute, “art is 
currently plagued by fraud, illicit business, and tax evasion” [1]. It is therefore easy to imagine 
how promises of transparency and accessibility brought forth by digital advancements would 
seem attractive to buyers who want to purchase art in good conscience. Buyers want to trust that 
“they are able to transact at a fair market price, that the piece is genuine, and that the seller has a 
legal claim to be selling it” [1]. The same is true for the reverse. Responsible sellers want the 
“ability to prove their ownership, [and] prove that the piece is genuine” [1]. Whether tainted by 
proximal association to WWII looted art, sourced from unknown or at-risk territories, or lacking 
sufficient records of provenance, authenticity, etcetera, there are innumerable scenarios 
necessitating responsible due diligence in art. The desire for trusted transactions from both 
buyers and sellers offers space for technologies to intervene and provide solutions. 


2.1 Due Diligence 


From issues of provenance and titleship to miscreant players in the industry, it is essential 
that buyers and sellers seek proper due diligence prior to any transaction. How does ‘proper due 
diligence’ look? At its core, due diligence is a risk management tool intended to protect all 
parties involved in a sale or transaction from engaging with illicit factors. These whether factors 
could be due to the object itself, the players involved, or the circumstances of a transaction. For 
example, in used car sales it is customary to research the car’s history and be able to answer 
questions such as: How many times has the vehicle been sold? Has the vehicle ever been 
involved in an accident or crash? What repairs have been made to the vehicle? Etcetera. Seeking 
answers to such questions is a form of protection for all parties involved in order to set a standard 
for the transaction. Furthermore, failing to know this information prior to purchasing a vehicle is 
seen as irresponsible. The same is necessary for art and cultural objects. Where was the object 
sourced from? Is there a clear record of ownership? Are there any red flags or grey areas 
associated with the geographical locations the object has been, or the hands through which it has 
passed? Committing oneself to seeking answers to these questions even though there is no 
governing body enforcing such actions is simply being responsible. This need for responsible 
market practices points back to the reoccurring question: what is needed to build trust in the 
market? 


Thankfully we are in a time where conversations around the possibilities of technology’s impact 
on the art market are forcing market participants to be increasingly mindful and aware of their 
actions. And this mindfulness is not limited individual potential buyers in the market. We can 
observe the way in which pan-national governing bodies as well value the creation and 
implementation of digital resources as well. On the 17" of January of this year the European 
Parliament passed a resolution calling for a “pan-European meta-database of looted-art, [and] 
funding for provenance research” as well as a proposal for “art market professionals to maintain 
a transaction register and provenance research programmes” [2]. 


2.2 Provenance Research and Databases 


Historically, provenance research has been limited to museums, libraries, and archives 
confining the practice to an academic setting. The increased media attention over the last decade 
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on the nature of fraudulent risks is helping to break the trend that due diligence research is kept 
to academia and is instead beginning to be implemented across the market. As observed in a 
report from the Smithsonian Institute, “digital technology is profoundly changing museums, 
libraries, and archives and the way they reach and interact with people...While it is true that the 
traditional role of our cultural institutions is still valued by the public and is needed, they must 
find ways to accommodate digital technology or risk becoming marginalized” [3]. Finding ways 
to implement digital efforts into art business and research practices is therefore imperative for its 
relevancy. And with the case of provenance research, digitization efforts and other technological 
schemes are breaking the mold of traditional archival research making it efficient and accessible 
to a growing audience. Individuals involved with digitization efforts of art historical resources 
have voiced a desire to see “better data capture methods, including mass digitization of archival 
and image materials to bring more of these “hidden resources” online” [4]. This desire has 
informed the creation of many growing databases. 


With over 1.5 million digital records of various art histories and information stored, The Getty 
Research Institute is encouraging the use of online databases and providing opportunities to do 
so via platforms such as their Provenance Index Databases. Similarly, law enforcement specific 
resources like the FBI Stolen Art File and the collaborative project “PSYCHE” between Interpol 
and the Italian Carabinieri Command for the Protection of Cultural Heritage offer searchable, 
digitized data of stolen art further adding to the landscape of digital resources easily used for due 
diligence research. Other growing databases such as the Artive Database, Art.sy, and the 
Archives for American Art just to name a few, are allowing information to be widely available, 
therefore encouraging collaborative rather than duplicative digitization efforts across the field 
[4]. It is our belief that through collaboration and an increasing presence of digitally accessible 
resources, provenance research and due diligence practices will become more and more 
promoted, hopefully one day becoming the norm for all art and cultural property transactions. 


2.3 Access to Pricing 


In addition to ownership history, some of these platforms are able to build trust within the 
market by sharing historical information on pricing. Pricing is an element that greatly attributes 
to the exclusivity and elitist quality of the art world, which various programs are beginning to 
challenge. What validates the price of an artwork? How are buyers able to trust that their money 
is being spent fairly? Applications and services such as ArtTactic, Magnus, and Pi-eX are 
committed to providing “financial clarity in the art world by mapping transactions...to give 
clients a better expectation of sales prices on artworks of the same artist by registering the sale 
prices of artworks” negating the exclusivity of often ambiguous asking prices [5]. These 
developments are important to our conversation because they further exemplify how technology 
is able to fulfill a demand and desire for art market participants — artist to dealer, buyer to seller — 
yearning to find their place within the landscape of the art market. Perhaps the main reason the 
art market continues to be opaque and highly consolidated is because the majority of players 
within it have been unable to find clarity within its practices. To alleviate this cloud of 
uncertainty, data and access to information is becoming progressively valuable and desirable. As 
observed by Pi-eX CEO Christine Bourrone, “art may not be an investment, but if you invest in 
it, you should be aware of the data” [5]. 
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2.4 The Challenges of Digital Research 


With such a wealth of information becoming accessible through these databases and 
various applications herein lies a challenge; how does a user determine which resources to use? 
While digitized records and searchable databases allow research to be efficient with a massively 
increased ease of access, the information on these various resources can be scattered and 
disconnected. A researcher could spend just as much time if not more navigating the extensive 
amounts of digital records as they might have spent visiting on-site research institutions. In a 
perfect world, a comprehensive, running list of databases with how-to instructions for each 
resource would exist allowing the process to be as unintimidating and inviting as possible. But 
through various studies, art historical researchers have expressed frustrations around “being 
unable to easily determine if tools might be relevant for their own work” and often claiming 
“they just don’t know what is out there” [4]. Additionally, it can be challenging to keep up with 
the advances and changes in digital resources, and with limited opportunities to formally learn 
best practices for digital research methodologies, researchers may unfortunately give up due to 
frustration without ever gaining an understanding that would benefit their research. 


2.5 The Technological Solution 


The magnitude of present and potential technological advancements, however, does 
provide a source of hope that could alleviate these challenges. The solution lies in a hypothetical 
but highly conceivable connectivity between digital resources. As digital assets are added to 
cloud-based technologies, they “can be linked to just about any other set of digital assets, 
offering a powerful tool for users to search for connections among not only museums, libraries, 
and archives, but also universities and information sources” [3]. In other words, through cloud- 
based technology and ‘digital-ledgers’ such as blockchain, large datasets can be linked, and 
sources fastened, allowing digital collections to be shared across platforms connected in the 
cloud. As evidenced by these challenges, the adoption of digital technologies in the art and 
cultural property sector has proven to be difficult and will continue to be so due to cost, and 
technical infrastructure. But through collaboration and a unified commitment among cultural 
institutions, art companies, independent researchers, and other market participants to embrace 
and explore the possibilities that technological advancements propose, successful integration is 
possible. Perfectly concluded in an exploration of the digital age in the cultural sector, the 
Smithsonian stated that “while digital technology poses great challenges, it also offers great 
possibilities” [3]. It is our responsibility to continue this conversation by exploring and applying 
technological advancements to instill greater transparency and wider participation in art business 
practices, research methods, and overall engagement of the fine art and cultural sector. 


3. The Big Picture 


What is the big picture? Digital technologies are changing the landscape of the art 
market. Digitization efforts are altering the methods in which research is conducted, and by 
whom. As our world becomes increasingly connected and information more accessible to a wider 
audience, the ability to pursue research outside the realm of academia is made possible. This 
presents an increase in the frequency of due diligence research opportunities, hopefully bringing 
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due diligence to the forefront of art and cultural property transactions. While technology’s 
impact on the art world is still fresh and there is a lot of work to be done before it has grounded 
itself in the market, the possibilities it presents are inspiring thought-leaders and participants. 
Through continued conversations and collaborative efforts, change is possible. 
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This paper describes an auditory and visual display of a population-based game, the 
Demographic Prisoner’s Dilemma. Agents’ parameters (such as life expectancy or strategy 
mutation) are gradually introduced to explore the population’s dynamics. As_ the 
implementation of the game undergoes such changes and variations, aesthetic considerations 
rise to the fore, prompting a re-evaluation of the system’s cross-space mapping and of 
computational aesthetics as a field of investigation. 


INTRODUCTION 


Evolutionary programming and algorithms have been used extensively for the production 
of works and generative systems in the domains of music, art and design. On the contrary, 
population dynamics based on a game-theoretical perspective have rarely been employed, to 
this end. Evolutionary game theory differs from other flavours of this area of applied 
mathematics in that it considers systems in which agents are not overly rational. Instead of 
analytical solutions to Nash equilibrium [11], evolutionary game theory is concerned with 
behaviours that are adopted or abandoned in the context of a population of such agents. 
Through repeated simulation of a Demographic Prisoner’s Dilemma [5], hereinafter DPD, the 
emergence of cooperation is observed, and consequently used for a conceptual blend [6] 
which maps cooperation to homogenous and harmonious audio-visual (AV) content. The 
blend between DPD and a population of AV agents is explored via modifying the cross-space 
mapping to obtain aesthetically pleasing results. Thus, questions concerning computational 
aesthetics or universal notions of beauty arise and are discussed. Arguing for a dialogical 
approach to aesthetics and a relational, dynamic negotiation of one’s artistic aspirational 
levels, future directions are considered. 


DEFINITION AND BLEND 


A DPD is an iterated Prisoners’ Dilemma [12], hereinafter PD, played pairwise amongst a 
population of agents who inherit a fixed (immutable) strategy and who are identical and 
undistinguishable from each other. A DPD differs from the classic PD in that it is a 
memoryless game, and agents do not get to change their strategy according to the recollection 
of previous actions or game outcomes. There are only two pure actions, namely cooperate (c) 
or defect (d). The game matrix for a PD is shown in Table 1, were the payoffs (rewards) for a 
given combination of actions are given in the corresponding tuple, with T>R>P>S. While in 
a one-shot PD game the Nash Equilibrium is the pure strategy dd, it has been shown [5] that 
in a DPD cooperation can emerge and endure. This is an exciting result which contradicts 
some intuitions and axioms of rational decision-making, and can inspire speculative 
explorations at a societal and artistic level. 
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c d 
c (R.R) (S.T) 
d (T,S) (P,P) 


Table 1: Prisoner’s Dilemma matrix. 


A typical game is such that a random number of agents is placed on a discrete grid, each with 
a fixed strategy. Each agent looks around its vision perimeter, choses a random neighbour 
within it, with whom it plays a PD game. Payoffs are added or subtracted to the agent’s 
wealth accordingly and, if such wealth exceeds a given threshold, the agent can reproduce 
within its vision perimeter. The child will inherit the parent’s strategy as well as a random 
portion of the parent’s wealth (which is detracted from the parent’s). Conversely, if an 
agent’s wealth falls below zero, the agent dies. 


Conceptual Blend 


To exploit this game at an AV level the notion of conceptual blend can be useful. This 
comprises two input spaces (which ones wishes to integrate), a generic space (which 
contains all the elements in common), a cross-space mapping (which contains connections of 
identity, transformation or representation), and a blended space (which contains all the 
generic structures as well as some that might be impossible in the input spaces). The 
projection onto the blendoid is a selective, non-unique process. A simple diagram for 
conceptual integration is shown in Figure 1. 


Cross-space mapping 


Figure 1: Conceptual blend. I1, I2: input spaces, G: generic space, B: blended space. 


The chosen cross-space mapping is now described. Each agent is represented as a coloured 
square in a discrete grid. The colour indicates the agent’s strategy: green for cooperate, red 
for defect, whereas the edge roundness is proportional to the wealth. Each agent is also 
endowed with a frequency modulation [3] process, where the carrier frequency is 
proportional to a fundamental frequency and the agent’s position on the grid. Each agent is 
thus a sound source and it is spatialised using binaural panning. The fundamental frequency 
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is different depending on the agent’s strategy. Defectors have a granular, noisy sound texture, 
whereas co-operators are partials of a harmonic texture which is richer as their number grows. 
This basic mapping was the first step of an ongoing exploratory process and, despite its 
simplicity, was able to render the population dynamics of the DPD. 


POPULATION DYNAMICS 


There can be many variations of the game and therefore of the population dynamics. These 
can be achieved by introducing changes to the properties of the agents. For example, agents 
might have (in addition to wealth and vision) the following parameters: maximum age and 
strategy mutation. The simplest version is such that each agent can live on indefinitely and its 
(eventual) offspring inherits its pure strategy with zero probability of mutation. In the 
simulations of this version, cooperation started to emerge around the 50th iteration. In the 
subsequent version of the game, a maximum age is set for all players. Above this threshold, 
an agent dies regardless of its wealth. Age is mapped to transparency, so that the older the 
agents becomes, the more transparent is shown. In this version, convergence towards 
cooperation seemed to emerge much sooner, around the 30th iteration. In addition to the 
maximum age, a third version introduces a probability that during reproduction the offspring 
of an agent might mutate strategy. This is the version of DPD where one can observe more 
variance in the population’s behaviour. In the epochs shown in Figure 2, the mutation 
probability was set to a value of 0.5, however, this probability value can be explored 
heuristically, to one’s liking. 


DPD - max age - mutation 


Figure 2: A plot of the co-operators/defectors ratio in three games of a maximum age, 
strategy mutation DPD. 


Beyond the effectiveness of the AV display as an aid to understanding the population 


dynamics of the DPD, the cross-space mapping needs to be considered further, since it is a 
crucial factor for obtaining aesthetically pleasing results. 
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AESTHETIC CONSIDERATIONS 


While the AV DPD described so far is sufficient to visualise and sonify the game’s 
evolutionary process, it is evident that the aesthetic result at a visual level falls short to being 
palatable (see Figure 3). To this end, it is useful to investigate further the notion of aesthetics, 
and in particular of computational aesthetics. 


Figure 3: A snapshot of a maximum age, no mutation DPD iteration. 
Computational Aesthetics 


Computational aesthetics is thought of as he theory, practice and applications of aesthetics 
in computing, however it has different connotations depending on the objectives and beliefs 
of who is considering it. For example, it “has been appropriated by engineers to mean the 
automated evaluation of image quality, by critics to mean the distinctive material genres of 
computer arts, and by technologists to mean specific programming technologies” [2]. 
Computational aesthetics often involve assumptions on universal notions of beauty, thus 
becoming liable of critical enquiry. This is because the broader field of aesthetics has 
developed beyond the theory of taste and beauty [9, 10], or of the philosophy of fine art [8], 
to include philosophy of criticism [1], art as experience [4] and pragmatist aesthetics [13]. It 
is unfortunate that the notion of aesthetics in computer science seems to have developed 
independently from that in philosophy of the arts. 

There are two main perspectives on computational aesthetics. While one investigates the 
value that computational processes or products have with respect to a human-centered notion 
of beauty, the other examines what it means for a computational system to be able to evaluate 
its own process and output. Thus, the latter is linked to some sort of machine meta-aesthetics. 
To this end, several aesthetics measures have been proposed and implemented in the 
computational domain, for example automated fitness functions or evaluation metrics based 
on loss functions. More in detail, these aesthetic measures can be broadly categorised into 
information and complexity based, geometric and fractal, psychological, biologically inspired 
and neuroscience based. Given the scope of this paper, the reader is referred to [7] for a 


46 


comprehensive survey of these measures and the methods used to implement them. We now 
turn to the perspective more akin to an aesthetics of digital arts, and how the AV DPD meets 
or violates the author’s goals and desires. 


AESTHETIC DYNAMICS 


To negotiate with the author’s aesthetic aspirational levels, a process of variation in the 
cross-space mapping was initiated. Given the exploratory nature of this endeavour, what is 
described below is only an initial step towards artistic speculations of the AV DPD. In line 
with the minimalist and meditative result at the sonic level (a continuously morphing 
harmonic texture where different partials are enhanced or suppressed based on the 
population’s behaviour), the colour gamma was dramatically reduced. Co-operators, 
defectors and background (empty nodes in the grid) were thus associated with very similar 
colours. Furthermore, the active cells of the grid were rendered with a varying stroke weight 
and spacing, proportionally to the wealth of the agents. The result can be seen in Figure 4. 

The main motivation behind these choices in the cross-space mapping was to arrive at 
smoother and more homogeneous results. While these changes were minimal, the results 
attained represented a step forward in meeting the author’s aesthetic expectations, and the 
system’s output can be considered sufficiently satisfactory. 


Figure 4: A visual variation on the DPD theme. 


DISCUSSION AND FUTURE DIRECTIONS 


In implementing suitable conceptual blends between evolutionary games and AV works 
the role of cross-space mapping is paramount. This is also the crux of other processes related 
to visual or auditory display, e.g., sonification of data. While it is unreasonable to seek 
aesthetic universals, it is important to set individual aspiration levels and to maintain a 
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dialogical process of continuous negotiation between the system’s designer and the 
computational process and output. 

Future work in this direction will likely involve further exploration of the cross-space 
mapping, and of the evolutionary scope. Agents’ properties, for example, could include more 
rules of behaviours, akin to cellular automata (e.g., The Game of Life) or a more varied 
visual (e.g., tints of colour) or sonic (e.g., oscillator type, audio parameters) character. Thanks 
to some of these variations, the AV process and result could become arbitrarily complex, in 
order to meet the designer’s criteria and objectives. 
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Abstract 


In this paper we conduct post-humanist media archaeological analysis of electrical 
network as media and investigate the digitalization of electrical network in one city. We 
present the cultural and historical contexts of the digitalization of electric networks, and 
argue that understanding the cultural and historical context of the digitalization of 
electrical networks is important especially when media archaeological theory seeks to 
understand the role of new ICT technologies used in our basic infrastructure. 


INTRODUCTION 


Media and administration both store, process and calculate information. In order to 
function, cities and administration also need hidden infrastructure, something that is 
now sometimes called the “grey media” or “logistical media”. In this paper we analyse 
electrical networks as media and provide an overview of the historical and cultural 
context of their digitalization. Electrical networks, sometimes also called as power 
grids, electric grids or electricity distribution networks are basic infrastructure of our 
society. Our approach to their analysis is post-humanist media archaeology. It was 
Foucault who shifted the emphasis from archives as dusty places where historians do 
research into more abstract conditions of knowledge. The technological dimension 
gained momentum informatics and in the so-called German media theory that has since 
then became successful German intellectual export. Media archeologist approaches the 
media technological assemblages both as an ethnographer who observes the interaction 
between humans and media and as an amateur engineer who experiments to learn how 
media technologies function as Kulturtechniken of information society. 


We mobilize media archaeology into an analysis of the media technologies of socio- 
cultural memory, analysis of the technical media infrastructures and associated the 
discourse networks comprising both human and non-human elements. Thus, our post- 
humanist media archeology is not limited to the archive or simple analysis of the 
traditional storage media. Instead, we focus on technologies of our media culture, its 
basic infrastructure and ultimately how media organises information in time. Our case is 
media archaeological overview of the electrical network of one Finnish city called 
Rovaniemi located in the Finnish Lapland, where publicly owned private entity called 
NEVE Ltd. provides and maintains services related to infrastructure, digitalisation and 
circular economy. We start our media archaeological overview by presenting cultural 
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and historical contexts of power networks as media and the digitalization of electric 
networks. Our analysis is based on media archaeological theory, historical documents, 
and ethnographic field research conducted in 2018. 


We focus on digitalization of electric networks because we want to see media history 
through ruptures. As a concept, digitalisation has been utilized to refer to the process 
where many areas of life are integrated into digital ICT-infrastructures. In theoretical 
debates digitalisation is often related to the questions regarding the role of information. 
In this case digitalisation changed the nature of electric networks from media 
archaeological perspective. Our main research question is: How digitalization has 
changed the electric networks from the perspective of media archaeological theory? 


Electrical networks as media 


This paper deals electrical networks through an engagement with media 
archaeological theory. We draw from media archaeological theory to better understand 
the digitalization process of electrical networks. The value of contemporary media 
archaeological theory is that it pays increased attention to the hidden technological 
infrastructure not commonly viewed as media. This allows to analyze technological 
infrastructures, devices and techniques surrounding us in their media cultural context. 


For us the most profound media theory in this paper emerged especially in Canada and 
West Germany after the Second World War. It was Harrold Innis [6] who understood 
the importance of infrastructure for media research and insisted that technical 
infrastructure should be at the heart of media theory. Another Canadian Marshal 
McLuhan [15] then popularised media studies that focuses on analysing how 
communication technologies are key agents in social and historical change. These 
media theoretical insights together with ideas from information theory and genealogical 
philosophical understanding were then picked up in West Germany. This led to the 
emergence of the so-called German media theory that soon became intellectual export. 
These classical media theories emerged as part of generation of other thinkers and 
reflection of political, technological, economical and media in that period. 


Kittler [8] wrote that the focus in media archaeology is on the “networks of technologies 
and institutions that allow a given culture to select, store, and process relevant data.” 
For Kittler, culture was already a procedure of data processing [1], and media were 
cultural techniques that allowed to select, store, and produce data and signals [14]. The 
production sites of data discourse networks emerge with technological media. They 
contain technological and institutional elements and can be analyzed by practicing 
technological media analysis that focuses on the operations of media structures in data 
processing — what Kramer [14] calls “the material ‘carriers’ of information”. In the 
context of mobile phones, which used to be very important field of industry in Finland, 
we could ask, what have been the forms of material carriers of information in mobile 
phones in the historical and cultural perspective and what could we learn from it in 
relation to media archaeology theories. 
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Kittler [9] seen networks of information in our basic infrastructure. Because cities have 
been “traversed and connected by a network of innumerable networks”, that 
regardless of conveying primarily information or energy, are all information, “if only 
because every modern stream of energy needs a parallel control network” (ibid., 718). 
When taking this logic to the end, cities are media because they calculate, store, and 
transmit information, and this was the basic definition of media for Kittler [9]. It is 
maybe evident that networks process, transmit and store information, but according 
to Kittler [9] this suffices to calculate all that is calculable. In other words, commands, 
addresses, and data make media abstract Turing machines, where the neural network 
based artificial intellectual deep learning algorithms hiddenly operates. 


In Literature, Media, Information Systems Kittler [10] points out that data processing is 
the process by which temporal order becomes moveable and reversible in the 
experience of space. This means that data processing for Kittler, storing is connected to 
spatial order, materializing a temporal process in a spatial structure [14]. It is the 
spatialization that enables media to capture temporal events [13]. In addition to Kittler it 
has been Wolfgang Ernst who has continued the analysis of timing mechanisms and 
time-giving technologies of technical media in German media theory. Ernst [3] argues 
that the essence of technical media is revealed in the processuality of temporal 
operations, resulting in timing (machine) agencies. 


Ernst’s discussion also moors the analysis of Kulturtechniken alongside media 
archaeology. According to Huhtamo & Parikka [5], media archaeology more like “a 
bundle of closely related approaches”, with a joint interest in the archaeological 
dimension of media technologies and media culture. For Ernst [2] the archive is a 
medium: “an economic place for symbol circulation, where a given present derived 
from numerous channels (“tradition,” “the past”) comes to light, to create a product 
called memory”. Media-memory archaeology then analyses what can be stored and 
transmitted in the apparatus that overplays the subject, writing about them transitively 
[16]. Ernst [2] concludes that now when all information may be stored, cyberspace is 
not only space but much more a topological configuration, where the storage procedures 
for multimedial space and the “anarchival signature “ is on display for media 
archaeologist. 


Ernst [3] summons that the concept of smart grid makes the electrical distribution 
network reactive in time-critical domain, allowing controllable logistical distribution is 
space and time. The basis of the time responses is now, like Ernst reminds us, the 
"immediate knowledge of the network itself". In practise, in case of electrical network 
interference digital time with 0 and 1 standstill there is no information flow, and not 
possibility for time manipulation, only immediate knowledge of the network exists. 
When the bits start to move and transfer data the time-space manipulation become 
possible. In 2007 electrical network in the whole city of Rovaniemi stopped for a few 
hours. The electrical network was not able to calculate, store, and transmit information. 
This practical case shows how central role electrical networks have as a media via 
communicating and transmitting information. Media now is broadly theorised as 
“assemblages or constellations of certain technologies, fields of knowledge, and social 
institutions” [4]. 
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In this paper we conceptualise the electrical networks as media; the main means of 
electrical networks as media communicate with the information society enlarge. The 
electrical networks and its infrastructure have fundamental communicative, informative 
and mobility role in modern digitalized world. Electrical networks are the platforms for 
organising information in our society. From the perspective of cultural history electrical 
networks, where the energy of charged elementary particles are formed, has been 
perceptualized as solid objects that could be seen and touched. However, digitalization 
has transformed electrical networks to invisible forms and fields of information, where 
electrical networks as media, as presented earlier, direct, transform, and in particularly 
illustrated dynamic change movements of our society. This makes media such as basic 
infrastructure an assemblage of associated technologies, fields of information and social 
institutions that maintain it. 


Digitalisation of electrical networks in cultural and historical context 


The historical and cultural overview of context of the electrical networks as media 
and their digitalization are investigated by illustrating from 100 years period of history 
electric network in the city of Rovaniemi located in the Finnish Lapland. The Imperial 
Senate ordered on October 21* in 1901 that the municipality of Rovaniemi should 
create new land use and building decrees for densely populated areas. This formed the 
legal basis of building electric networks in the city area by providing autonomy that was 
the beginning point of electrical networks in the city of Rovaniemi. Electricity 
distribution in city of Rovaniemi began 1914. First there was 1000 pieces of lamps 
connected to electrical network by individual citizens and on May 25", 1914 there was 
38 street lamps installed to the city. At the end of the year 1914 flux consumption was 
1300,88 kilowatt-hour. In 1925 there was already 7259 lamps in the compact city of 
Rovaniemi. In 1940 the demand for electricity per year was already 4 million kilowatt- 
hour. [7] 


In 1944, near the end of the Second World War, almost complete destruction of the city 
Rovaniemi in the phase 4 of the operation Birke (“Birke zerkleinern”) demolished the 
electrical infrastructure of the city. The decades after the war were the decades of 
rebuilding, development and rising the standards of living. In 1950 electricity volume 
increased fivefold and demand for electricity in 1960 was three times higher than 1950. 
In both the 70s and 80s, the volume of electricity distribution doubled. In the 90s the 
electricity consumption has been increasing somewhat slower. The digitalization history 
in electrical networks also modify concepts in theoretical and practical level. In 2000 
the electrical networks in Rovaniemi commenced to be described as smart grids, 
where data transition, digital processing of applications and information management 
via user interfaces occurred. The interfaces within the tens of applications in NEVE Ltd. 
were built to be able to control data flow, network security ja information. In 
digitalization process electronical networks, the role of information management and 
organization in time has become core. Operational manager at NEVE Ltd. describes 
digitalisation process in practical level: 
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Until the year 1986, all the machineries in electronical network 
were manual. First phase to digitalisation process was so called 
Procol-measurements equipment, which functioned via 
telephone networks and were used to boundary point 
measurements. Second phase was in year 1995, when the 
electrical meters were measured via computer data networks. 
However, only the biggest electrical consumers were measured 
via network. In 2012 all the kWh electrical meters were read 
remotely via radio/telephone network. In this phase, in addition 
to kWh consumption also, the data from malfunctions of low 
voltage interference alerts was transferred remotely and could be 
analysed from graphical user interface. (... ) Also, the security 
of electronical networks increased in 2000 and the science of 
cryptology become more and more important. In electrical 
networks, the machines can be programmed the proper security 
level. Regarding to maintenance, the sensors in electrical 
network transfer data for malfunctions. Electrical network is 
quite stable, there is only 50 malfunctions cased per year. 
(Interview of the operational manager) 


Operational manager highlights the role of measuring electricity consumption remotely 
and the role of ICT in risk management. However, some of the people encountered 
during in the field during our ethnographic research also reflected the broader role of 
the electric networks. The chief of the digital business services in NEVE Ltd describes 
the implications of digitalization for the electric network: 


In practice, in second millennium electronic networks were 
started to see as part of media, information and communication 
technology. NEVE Ltd. [Rovaniemi energy corporation] own 
local area networks via ethernet cables that were provided for 
citizen in Rovaniemi first time in 2018. The electronic networks 
were seen as a new form of media inside and outside the energy 
corporation, where in LAN-networks different applications, 
massive data flows, security and information and 
communication technology were organizing information in time. 
The new era of digitalization has become. [...] The most 
important thing is to control complex information flows in local 
area networks. (Interview of the chief of the digital business 
services) 


Much of the work that has been going on in electric network digitalization, also 
substation and distribution automation, is now included in the general concept of the 
smart grid. In the following picture engineer is operating the electrical substation before 
digitalization (see picture 1.). In the digitalized system the same information can be seen 
in digital user interface (figure 1.). This illustrates the cultural and historical shift caused 
by the digitalisation. 


53 


(04 e - , t 
Picture 1. Pekka Kauppinen, C 
operating VII sa in physical form in the 70's. (located in Viirinkangas) 


Babio ied yie pott Chip Vito ipto 
a» PROSESSIKUVA - VITRINKANGAS SA A BA FELL P 
PETE eee Rh”. 


BO Ga Sert foe. Y4 HOV camerata sumo SILICIO NANA SA 


E saco 

© mm © nno 
@ wm © mm 
@ iris @ non 
@ mam © un 


E saon 


@ msm @ tanto 
@ min © nmo 
mé 729 1210 mé 7510 1526 Hi. mọ ws me O mm è @ mmn 


161 7620 mÒ mé må 7026 7916 1920 71030 1020 nné 2026 uag 120 714) © 720 nse 72520 nað mag 10716 7472. ua O 71526 7293.0 21926 72010 1970 m0 12126 


4. 
whe whe whe whe me fe a LI vede fs fia fe vee me aa afa 
$ ha n date xi = 
7990 nos mò 1m9 149 ns 169 mms 189 Ò 7199 
eee a MA OO SOS SEI EY 


Figure 1. Electrical substation in digital user interface 2018 (located in NEVE Ltd’s 
main building) 
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Rovaniemi energy corporation own local area networks via ethernet cables were 
provided for citizen in Rovaniemi first time in 2018. This illustrates that the digital 
transformation of the basic infrastructure like the electrical network can be arduous and 
slow. Based on our analysis, in our study the electrical networks is seen as medius, 
within its intermediate functions in IoT network. The electrical networks as media shall 
more and more organize the information in time in our society, in which we have 
become reliant. 


CONCLUSION 


Electrical networks have slowly been digitalized. They serve as our basic 
infrastructure and as media communicate with the information society enlarge. There 
have been ideological debates concerning the role private sector in the maintenance of 
basic infrastructure in monopolistic situation that guarantees steady dividend payments 
for the investors. Nevertheless, we often notice and pay attention to the communicative 
nature of our basic infrastructure when the technology breaks. When there is no 
electricity the urban mobility soon becomes very complicated and logistic systems of 
contemporary economy halts. Especially in the urban settings digitalization not only 
allows possibilities for diagnostics that helps in maintenance, but also communicates 
with the human agency and other technical systems that consume electricity. 


The benefits of analyzing the digitalization of electrical networks as media is that 
media theoretical approach reveals the role of hidden infrastructure and highlights its 
communicative function. Like Durham Peters (2015, 4) observes, “[d]igital devices 
invite us to think of media as environmental, as part of the habitat, and not just as 
semiotic inputs into people’s heads.“ This environmental view of the media is in the 
heart of the smart city thinking, “where data is no longer produced through statistical 
apparatus” but instead with networks dedicated to capturing data often using space 
technology “that are an advanced and geostationary stage of exosomatization, more or 
less specialized, functioning as relays for networks of all kinds” [18]. Building and 
digitalising basic infrastructure is resource-intensive and can take time, and in our post- 
humanistic media archaeological study we have not only analyzed the electrical 
networks as material remains of human life and activities, but described the cultural 
historical of digital transformation of electrical networks as media. This environmental 
view of the media is vital when media archaeological theory seeks to understand the 
role of new ICT technologies used in our basic infrastructure. 
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Abstract — In order to still be able to induce emotions and to increases people’s awareness, artistic 
production evolves. In our complex and liquid world, multidisciplinary and technology innovations 
are part of this transformation. From the experience of the multidisciplinary artistic group O3LAB, 
we created “It’s time to give back”, an installation that addresses the problem of ecology using media 
and video art and interaction design. 


INTRODUCTION 


In last decades technological and economical innovations led to some important consequences in 
the way we behave. We live in complex and liquid world (Bauman, 2000). In this complexity we 
are discovering that solutions to problems require to have a broad knowledge to address all the 
facets and to evaluate all the consequence. However, on the other side, we are going towards a 
professional hyper specialization. This brings the need of a continuous interaction between 
disciplines: at all levels more and more organizations discover the importance to adopt 
multidisciplinary working teams (Agile Paradigm). 


Technology is one of the fundamental actors of the human evolution since nowadays it has a great 
role in our daily life: among the other effects, it deeply transformed the way we communicate, and 
we build relations. 


In this scenario, how has the artistic production changed, or is changing? How can art still induce 
emotions and increases people’s awareness? How can art still be the “lie that reveals the truth”? 

To find the answers to these questions is a hard task, but we highly believe that interdisciplinary 
(Ian Heywood, 2012) and the adoption of technology not at support to the creative process, but into 
the creative process (Alexenberg, 2011) are two of the key elements in the transformation process 
for the present and the future in artistic production. 


In support to this vision, we founded O3LAB, which is an artistic group composed by technologists 
and artists in different disciplines. O3LAB created “It’s time to give back”, an installation that 
addresses the problem of ecology. The installation is divided into two distinct sections: a video (in 
loop) and an interactive installation. The video is an accusation of our lack of interest to the 
environment, we have little time to change (maybe we do not have any more time) and therefore 
"it's time to give back" what we took from nature. Following we show some frames of the video. 
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The interactive installation attempts to push the attenders to think at the need to move to action 
through and embodiment paradigm thanks to interaction design techniques. 


“GIVE IT BACK” AN INTERACTIVE INSTALLATION 


Ecology is one of the most on-the-spot and controversial problems that involves the entire world at 
different levels. It is controversial for several reasons. Among them we believe that two specific 
aspects have an important role: 
do not exists a unique single solution — do not exists a unique solution to the problem, 
more likely the solution will be the consequence of our personal positive or negative actions 
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slow changes - the effect of our personal positive or negative actions have only a small 
impact on the entire system, but the contribute of a mass of people can be highly be visible 
after some time of constant contribute 


“Give it back” is an artistic installation through which attenders can interact with the artistic 
subject. The interaction paradigm allows people to make a personal and immersive experience that, 
through the embodiment paradigm, drive them to go in deep into the ecological problem and to 
increase the awareness of the need of a personal contribute. 


The main subject of the installation is a virtual tree that evolves as long as the visitors interact with 
it. At the beginning the tree is presented as bare and dry. Through and interaction process the 
attenders have the change to contribute to reflourish it. Figure 1 shows the virtual tree in three 
stages of its evolution: the bare and dry tree at the beginning, a semi-reflourished tree and the 
reflourished tree. 


Figure 1 - Snapshots of three stages of the evolution of the tree 


The visitors, one at the time, through specific gestures are able to generate an animated virtual drop 
of water that will appear on the same projection plane of the tree (Figure 2). 


Once the drop is generated, its movements will be synchronized with the 
visitor’s hand movements and the visitor will be able to move it through 
the screen. A vertical movement of the hand in the space corresponds to a 
proportional vertical movement of the drop and a horizontal movement of 
the hand in the space corresponds to a proportional horizontal movement 
of the drop on the screen. With a forward movement of the hand, the user 
will be allowed to feed the tree bringing it the drop of water. Once the 
action is complete some few leaves will grow on the tree. 


Figure 2 - Virtual drop of water 
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The interaction is design in order to embody the two aforementioned elements (do not exists a 
umigile single solution and slow changes): 
Each one is allowed to concretely feed the tree. This is to underline that each one can 
contribute through a concrete action. 
The attenders need to understand how to give their contributes through the systems, indeed 
no instructions are given so that each one can discover how to interact with the tree. This is 
to underline that each one needs to find their way to provide a contribute to solve the 
problem. 
The attenders are free to choose if they want or not to give the contribute, they are allowed 
to just play with the drop without feed the tree. This is a metaphoric way to express that 
each one is free to choose if he/she wants to provide his/her own contribute or not. 
The attenders are free to choose the number of contributes, this is to underline that we are 
free to choose how much we would like to be involved in the problem. 
Each contribute has a visible but small effect on the re-flourishing process. Just few leaves 
will appear at each contribute, this is to underline the slow changes issue. 


We realized “Give it back” for the first time at the 5°Creativity and Design Week in Chengdu 
(China) in a dedicate booth shared with External Reference Architects’ and Sporph. In Figure 3 we 
show an overall picture of the interactive installation: it is well visible the projection of the tree and 
the drop and the sensor used for the hand tracking. Figure 4 shows an overall view of the 


installation into the dedicated booth. The projection of the tree occupies the entire bottom wall of 
the booth. 


Figure 3 - View of the interaction Figure 4 - Overall view of the interactive installation in the booth 
paradigm 


1 http://externalreference.com 
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TECHNICAL SETUP 


Figure 5 -Overall schema of the system 


Figure 5 shows the overall schema of the system. Two are the main components: the projection 
and the tracking system used for the interaction. 


As far as projection is concerned it consists of two layers projection. The first layer concerns the 
tree and the second concerns the drop. It is interesting to mention that the tree has been modeled by 
using the 3d modeling software SpeedTree? mainly used in movies and games industries. 


The interaction system is based on the tracking of the hand of the attenders. It is realized through a 
Leap Motion hand tracking system’ placed in front to the projection as shown in Figure 3. 


Leap Motion controller, through two cameras and three infrared LEDs, is able to provide a 3D 
reconstruction of the skeleton of the human hands (Figure 6). The system allows to track various 
joints and output the positions and the rotation for each one of them. The controller is able to 
acquire data at 60 fps as the frequency rate. 
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The high data acquisition sample rate and high data transmission rate guarantee the real-time 
experience to the users that do not perceive any temporal lags from the movement of their hand and 
the movement of the drop on the screen. This is also guaranteed by the optimized. 


Each time it receives now data from the Leap motion controller related to each joint, the 
coordinates adapter software infers the average position of the hand and transposes it from the 
spacial coordinate system adopted by the controller to the coordinate system adopted for the screen 
projection. 


The software has been implemented using OpenFramework, a C++ framework for media art and 
interaction design. The C++ language is suitable for applications were low computational time is 
required. Our software has been also optimized to guarantee the real-time interaction. 
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Abstract — Public administrations need digital archives that are typically maintained 
as centralized repositories or databases. This solution constitutes a security and trust 
single-point-of-failure. We have faced with these issues in the case of digitalization of 
the General Catalogue for Cultural Heritage which gives support to promotion and 
safeguard of the Italian national heritage. In this work we propose a blockchain-based 
architecture for that digital archive that guarantees security and availability in a 
decentralized, efficient and cost-effective way. Furthermore, the architecture gives the 
chance to restore the catalogue contents in a tamper-proof way. 


I. INTRODUCTION 


Public authorities collect and use a large amount of information in the form of digital 
archives which are mostly implemented as centralized systems. In case of a collection of 
structured, interrelated data, a digital archive is maintained as a database. 

Centralization raises several security issues. First of all, it implies that customers trust 
organizations to effectively keep data secure. At the same time, it makes public authorities 
prone to attacks because it is the authority itself that controls the archive. Furthermore, a 
centralized database system represents a single-point-of-failure. This was particularly 
common in Web 1.0, where entities setting up their own servers. Things did not get much 
better in Web 2.0. Cloud providers offer cheap, sometimes even free, scalable database 
services (DBaaS), but scalability and fault-tolerance come at the cost of providers’ owning 
the content, profiling users and collecting huge amount of data about user activity. 

We faced with the above issues when we considered the “Catalogo Generale dei Beni 
Culturali” (General Catalogue for Cultural Heritage), hereafter the “catalogue”, under the 
control of the Ministry of Cultural Heritage and Activities and Tourism [8]. This database 
plays a significant role in the spreading of knowledge and in the promotion of the Italian 
national heritage. Furthermore, it contributes to manage crimes and natural disasters that may 
affect the national heritage. 

In this work we propose a blockchain-based architecture for the catalogue. For performance 
and cost reasons, we also adopted an off-chain data storage strategy. 

There are several reasons for overall solution. First, it is architecturally and politically 
decentralized, so overcoming the security and trust issues previously mentioned. In addition, 
it is secure as it ensures integrity, long-term data availability and authenticity. In order to 
guarantee some degree of privacy, data is also encrypted, whenever required, at the 
application level. The properties we have just presented are of paramount importance for the 
catalogue. It is important to outline that the proposed solution gives also the chance to restore 
the catalogue contents, as needed, with the guarantee that data has not been tampered with. 
Finally, the solution is also efficient, both from an economic and performance viewpoint. 
The remainder of the paper is organized as follows: in Section II we discuss related work. In 
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Section III we introduce the catalogue and explain the way we created a (reduced-size) clone. 
In Section IV we describe the proposed solution. In Section V we present an early prototype. 
In Section VI we evaluate the cost of our solution. Finally, Section VII concludes the paper. 


II. RELATED WORK 


The problem of managing records through a blockchain has been largely investigated 
during the last few years. In her paper, Lemieux proposes a classification of blockchain 
applications [1], based on which information is stored in the blockchain: a) mirror type, b) 
digital record type, c) tokenized type. 

In the mirror type, the blockchain serves as a mirror, which stores only records fingerprints. 
The complete information of a record is stored into an external repository and the blockchain 
is used only to verify records integrity. In [2] the authors describe a first implementation of a 
decentralized metadata database, based on the combination of the blockchain and IPFS 
technologies. In their paper Liang et. al. describe ProvChain [3], a system which guarantees 
data provenance in cloud environments. Vishwa et. al. [4] illustrate a blockchain-based 
framework, which guarantees copyright compliance of multimedia objects by means of smart 
contracts. 

In the digital record type, the blockchain is used to store all the records in the form of smart 
contracts. In [5] the authors illustrate a distributed and tamper-proof framework for media. 
Each media is represented by a watermark, which is firstly compressed and then stored into a 
blockchain. Approved modifications to media are stored in the blockchain thus preventing 
tampering. In [6] the authors describe Archain, a blockchain-based archive system, which 
stores small-sized records. Multiple roles are defined in the system, thus allowing records 
creation, approval and removal. 

In the tokenized type, records are stored in the blockchain and they are linked to a 
cryptocurrency. This constitutes an innovative case, where the literature is not consolidated 
yet. Adding, updating or removing a record has a cost. An example of this type of blockchain 
is represented by the Ubitquity Project [7], which records land transactions on behalf of 
companies and government agencies. 


III. THE SAFEGUARDING OF THE CULTURAL HERITAGE 


The General Catalogue for Cultural Heritage [8] is a centralized database system which 
collects data about Italian cultural assets. It responds to protection and promotion purposes of 
the national heritage by providing users with the knowledge of Italian cultural works. 

The catalogue is aimed at identifying and describing all cultural objects for which a specific 
artistic, historical, archeological or anthropological interest has been recognized. 

In particular, the catalogue allows to consult information related to thirty different typologies 
of cultural assets. Each typology is properly described by means of a specific standard, called 
“scheda di catalogo” (catalogue record), whose definition is led by the “Centra! Institute for 
Cataloguing and Documentation” (ICCD). In more general terms we can state that a 
catalogue record is basically a set of attributes which takes into account peculiarities and 
intrinsic key features of each typology of cultural object. 

Around May 2016, the ICCD has launched a project, called Openiccd [9], which aims at 
sharing, among other things, raw data about cultural assets being catalogued. Open-data are 
available for each Italian region and refers to the following two typologies: “archeological 
exhibits” and “artworks”. 

Initially, for the sake of simplicity, we have limited ourself to the ICCD open-data related to 
exhibits and artworks of Tuscany which however amounts to 75.662 records with 51.977 
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images in the form of URLs and 46 attributes for each record. 

In order to store these data, we chose a NoSOL database system. In particular, we selected the 
Elasticsearch "document store"-type database [10]. In Elasticsearch, any information item is 
stored into an index which is a collection of documents that have similar characteristics. 
Therefore, for the storage of Tuscany ICCD open-data we defined two indices, one for 
artworks and the other for exhibits. Before being stored into Elasticsearch, open-data have 
been parsed exactly as defined in related catalogue records. 


IV. A BLOCKCHAIN-BASED ARCHITECTURE FOR THE 
SAFEGUARDING OF THE CULTURAL HERITAGE 


With reference to Figure 1, the solution we propose assumes a frontend that allows 
principals to insert, edit, delete or search cultural assets. We assume these principals do not 
behave maliciously and thus are trustworthy. The designed architectural solution assumes 
also the following components: Elasticsearch, the Ethereum blockchain and the 
InterPlanetary File System (IPFS). 

When the frontend issues a search operation, the execution of this operation only involves 
retrieving data stored in a specific Elasticsearch index. In contrast, in case of a modifying 
operation, i.e., an insert, update or delete operation, things are slightly more complicated. 
The operation is reflected on the local database, i.e., Elasticsearch which simply acts as a 
collector where information about cultural objects is stored and retrieved by. After that, the 
Ethereum blockchain [11] comes into play. 

We have considered Ethereum because it is a public and Turing-complete blockchain. 
Furthermore, by means of an Ethereum smart contract, we let the catalogue operate 
programmatically without the need for ownership or control by a particular entity. In this 
sense, an ad-hoc designed Ethereum smart contract has been provided. The smart contract 
exposes a method which fulfils a twofold task: a) to check whether or not the operation has 
been issued by an authorized trustworthy principal; b) to store the operation and an identifier 
for the Elasticsearch index involved in the operation itself into the blockchain. As to point a), 
when a modifying operation is issued at the frontend, a principal is always required to sign an 
Ethereum transaction. When the transaction is signed, the smart contract method is invoked. 
At that time, authentication is performed. If the principal is indeed authorized, the operation 
and the identifier for the Elasticsearch index are stored into the blockchain. However, 
because Ethereum on-chain accounting has different limitations, namely, storage of data is 
really expensive and withdrawing data is relatively slow, the operation has actually been 
accounted off-chain. 

Specifically, a JSON object encoding the operation and the related arguments, i.e. 
information about the item, is created. The JSON data-interchange format has been chosen 
because a document, the basic unit of information that can be indexed in Elasticsearch, is 
expressed in JSON. Off-chain means the JSON object being stored into the content 
addressed, versioned, peer-to-peer, distributed file system IPFS [12]. At that time, an JPF'S 
link under the form of cryptographically secure digital digest is obtained and stored on-chain 
together with the identifier of the Elasticsearch index that gets modified. 

The execution of a modifying operation takes place in six steps. 
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Figure | - Tasks carried out by the frontend in case of insert, update or delete operation; a 
dashed arrow indicates a conditional task. 


1. If the modifying operation is either an insert or an update operation, the frontend may 
encrypt some of the information about the item as needed; 


2. The frontend executes the operation on Elasticsearch. Moreover, the frontend keeps 
track of the specific Elasticsearch index involved in the operation. 


3. The frontend creates a JSON object encoding the operation and the related arguments. 
In the case of an insert or update operation, the encoding complies with the rules 
defined in the item related catalogue record. 


4. The frontend stores the JSON object into IPFS. IPFS returns an IPFS link to the 
frontend. 


5. The frontend prompts the principal to sign an Ethereum transaction. 


6. As soon as the transaction gets signed, the frontend invokes the Ethereum smart 
contract method. The method checks if the signer belongs to the authorized ones. If 
this is not indeed the case, the transaction is rejected. Otherwise the transaction 
continues and the following two pieces of data are stored into the Ethereum 
blockchain: a) the permanent and immutable IPFS link; b) the identifier specifying the 
Elasticsearch index involved in the operation (tracked at point 2). 


Let’s make some comments about our proposal. First, the proposed solution actually 
overcomes all the problems described in Section I. In particular, Ethereum guarantees the 
integrity of IPFS links and Elasticsearch identifiers while IPFS guarantees the integrity of 
JSON objects. 

In addition to integrity, the solution provides also availability for two reasons. First, the IPFS 
infrastructure guarantees availability of JSON objects, provided the infrastructure is properly 
designed, that is its contents are replicated over many nodes. Second, Ethereum guarantees 
the availability of IPFS links. 

Finally, it guarantees the authenticity of the catalogue. This is because the smart-contract has 
been conceived to allow only authorized trustworthy principals to issue modifying operations 
and thus generate and store IPFS links. 

Actually, in designing a cost-efficient solution, we have gone far beyond the off-chain 
storage approach. When talking about costs, we implicitly refer to gas, that is the key 
mechanism underlying the security of Ethereum. In this respect, it is worth focusing on the 
smart contract implementation. As mentioned, the smart contract consists of just one method. 
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The method takes the IPFS link and the identifier as parameters in order to store them. To 
actually accomplish this task, we have enforced the body of the method to cause the 
Ethereum Virtual Machine (EVM) to emit an event. The specific event has also been 
provided within the smart-contract implementation. The arguments for the event call are 
indeed the IPFS link and the Elasticsearch index themselves. 

In general, when an event is emitted, a log is generated. A log is a data structure which is 
available in the receipt associated to any transaction. We came up with the idea of storing 
IPFS links and identifiers into Ethereum logs, being a log the cheapest kind of storage. 
Furthermore, since generating it requires to pay (among other things) also 8 gas for each byte 
of data, we designed two complementary strategies. Data is actually intended as the 
arguments passed to the event. As previously mentioned, the arguments passed to the event 
are the IPFS link and the identifier. The first strategy enforces an IPFS link to always be 
converted from the default 46 characters base58 multihash encoding to a 32-bytes base16 
encoding. This allows us to use a bytes32 data type instead of a string datatype for the IPFS 
link. This requires less gas; hence it costs less. 

As to the second strategy, instead, we let the data type for the identifier be an uint8, i.e. just 
1-byte. The reason for using specifically an uint8 is the following. In the most general case, 
we have just as many Elasticsearch indices as the number of possible typologies of cultural 
objects (30). A possible solution consists in defining a lookup table where typologies are the 
table keys while values are the identifiers codified by unsigned starting from 0. Using an 
uint8 type for the identifier, which was more than the required, resulted also in saving gas. 
To conclude, it is worth noting that the proposed solution gives also the chance to restore the 
catalogue contents wherever necessary. This may be actually required because forgeries, 
thefts, natural disasters or even because some nodes in the Elasticsearch cluster are faulty. In 
all such scenarios, there is a strong interest in finding out the last consistent state of all the 
cultural assets affected. In this respect, it is not appropriate or even not possible to exploit 
information about the items stored in Elasticsearch. This is because nodes may be 
compromised or crashed respectively. In the former case in particular there is no guarantee 
that information about the item has not been tampered. 

The solution we propose overcomes this limitation. Moreover, having stored in an Ethereum 
log also the identifier for the specific Elasticsearch index involved in a modifying operation, 
we are actually able to restore only specific Elasticsearch indices, that is just specific 
typologies of cultural objects. 

With reference to Figure 2, restoring an Elasticsearch index consists of the following steps: 
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Figure 2 - Restore an Elasticsearch index. Overview. 
1. The user specifies at the frontend which Elasticsearch index(indices) to restore, that is 


which typology(ies) of cultural object(s). The frontend exploits the lookup table in 
order to get the correspondent 1-byte identifier(s); 
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2. The frontend gathers Ethereum logs from the blockchain and filters them based on the 
value(s) of the identifier(s). To this aim, we have declared the identifier, in the smart 
contract event signature, with the reserved keyword “indexed”; 


3. For each gathered log, the frontend turns the 32-bytes IPFS link (stored inside the log 
itself) back into the base58 encoding. At that point, the frontend uses IPFS links in 
order to retrieve from IPFS the correspondent JSON objects; 


4. Finally, the frontend executes operations encoded in retrieved JSON objects exactly in 
the same order they have been initially performed. 


V. AN EARLY PROTOTYPE 


We implemented an early prototype which comes in the form of an Ethereum 
decentralized application. Differently from the conceptual description in Section IV, the 
actual implementation of the backend code actually consists of three (3) smart contracts 
written in Solidity. They are three instead of one, because each one provides exactly one of 
the functionalities described in Section IV. This is a typical approach so as to provide 
modularity. 

A smart contract named “AccessRestriction” allows only an authorized principal, 1.e. the 
creator of the smart-contract itself, to store IPFS links and identifiers. We have adopted this 
behavior, instead of a whitelist of authorized principals, just for the sake of the prototype 
itself. To this aim, the contract implements a “restriction access” pattern by means of an 
“onlyvOwner” Solidity function modifier. 

The “CulturalGood” smart contract actually encodes the business logic of the application. 
Finally, another contract called “Mortal”, which inherits from AccessRestriction, accounts 
for the specific need of removing the code and storage from the blockchain. This may be 
required for whatever reason, at a point in time. Actually, the CulturalGood contract inherits 
from Mortal, which in turn inherits from AccessRestriction, hence it actually provides both 
the two functionalities previously described. 

The backend code has been deployed and run on a private and local Ethereum blockchain 
named “ganache-cli”. With respect to the client side, instead, we implemented the frontend as 
a Web application running on Google Chrome. The Web application is written in HTMLS, 
CSS3 and JavaScript. The jQuery library has also been used. 

The next component is “Metamask”. It allows us to connect, from within the Google Chrome 
browser, to the custom RPC endpoint exposed by ganache-cli. In this way, it is possible to 
import ganache-cli defined accounts and sign transactions. The last two components are the 
IPFS service and the Elasticsearch service. 

The prototype we just presented has been deployed within a containerized dev-environment 
with Docker [13]. Specifically, a multi-container Docker application has been built. 


VI. COST EVALUATION 


To evaluate the cost of our solution we need to consider two costs: a) the cost for the 
deployment of the CulturalGood smart contract, and, b) the cost related to insert, update or 
delete operations. 

With respect to the first cost, the contract creation transaction accounts for 289154 gas. The 
latter is due to the cost of several EVM operations that the transaction affects. 
The cost for the contract deployment has been evaluated on 16 October 2018. Specifically, 


69 


considering a gas price equals to 10 Gwei (< 2 minutes confirmation time) and the exchange 
at that specific time (1 ETH = 183 EUR), the actual cost for the contract deployment was 
approximately EUR 0,53. 

As to the operations cost, instead, when a new cultural object is inserted, or when an existing 
one is altered or deleted, as we already mentioned, the user gets prompted with a (Metamask) 
notification in order to sign the transaction. When the transaction gets signed, it is then 
broadcasted. Broadcasting the transaction causes the CulturalGood smart contract method to 
be invoked. The function call once compiled into transaction data, calldata, accounts for 68 
bytes overall. The first 4 bytes represent the method’s signature followed by parameters in 
chunks of 32 bytes. In our case, the parameters are such that the first 32 bytes encode the 

IPFS link whereas the subsequent encode the identifier. Whatever the operation performed, 
calldata is such that the method's signature is always the same, what can vary are only the 
parameters. In the worst case, in terms of gas consumption, we end up with only 31 bytes out 
of 68 being zero. These corresponds to the 31 most significant bytes of the identifier. 

So, we can compute the upper bound of the amount of gas to pay. It turned out that, in the 
worst case, the transaction accounts for 25642 gas. Following the same reasoning as above, 
the cost for storing within a log an IPFS link (32 bytes) and an identifier (1 byte), in the 
worst-case amounted to EUR 0,047. 

It is worth noting that costs we have just presented refer to the prototype implementation. In 
practice, in order to allow a whitelist of authorized principals the costs will be higher. 
Finally, the actual overall cost will depend on the number of insert, update and delete 
operations that will be performed over time. In this regard, we try to give an estimate based 
on both the information available in last ICCD public survey [14] (2011) and the total 
number of cultural assets actually stored (~ 2,7 Million). According to the just mentioned, 
every eight years roughly 1,35 Million cultural assets get indexed. If we suppose the same for 
the next eight years (2019-2026), i.e. considering only insert operations (plus considering the 
same exchange rate as before), the cost to pay will be EUR 63.450. 

The ICCD survey also presents the total expenditure for cataloguing from 2002 to 2009. This 
amounted to EUR 12.495.757. The estimate of EUR 63.450 actually accounts only for 0.5% 
of that cost. If we consider the additional value provided by the solution, it may be 
worthwhile for the ICCD bearing this cost. 


VII. CONCLUSIONS & FUTURE WORKS 


In this work, we have described a blockchain-based architecture for the safeguarding of 
the cultural heritage. The proposed solution is decentralised, secure and efficient, both from 
an economic and performance viewpoint. The solution also allows to restore the catalogue 
contents in a tamper-proof manner where necessary. 

Finally, it is right to note that in our prototype JSON objects are supplied to one single IPFS 
node only. Even if data cannot get garbage collected, since automatically pinned after being 
added to IPFS, we still have data on one single IPFS node. In order to improve the data 
redundancy, for the reasons outlined in Section IV, one possible enhancement may be 
keeping several IPFS nodes online pinning the data. The more the IPFS nodes pinning 
content, the better the redundancy. 

In this respect, there are some tools being developed which can help in sharing the costs of 
bandwidth and disk-storage. Among all possible implementations /PFS Cluster seems to be 
the most promising. One possible future work may involve designing a solution of this kind. 
Another possible future work may involve designing a solution in which a whitelist of 
authorized trustworthy principals is allowed to store IPFS links and identifiers. 
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The light in addition to being the factor that most influences the human perception, makes a 
leading wellbeing role in everyday life. Learning about relationships, influenced by the light, 
amongst emotions and behaviors, is a relevant investigation to stimulate applications and 
projects that can help us to improve the health and emotional status. Through mental processes 
related to our individual experiences, we attribute to light the ability to make more familiar the 
environment that hosts us. Feeling at ease in a place, it depends much of our relationship with 
light. The perception of the visible that enables us to understand and interact comfortably with 
everything that surrounds us, taking, even unconsciously, the well-being from the pleasure to be 
a part of that place and to appreciate its characteristics without efforts: the niceness of enjoying 
and be enjoyed. 


Light, besides being the factor that most influences human perception, contributes in a pre- 
eminent way to the psychophysical well-being of the individual in everyday life. 

Knowing relationships, influenced by light, between emotions and behaviors, is a relevant 
investigation to stimulate applications and projects that improve health and emotional state in 
an environment. 

In perceiving one's relationship with space, each of us needs to make sense of the surrounding 
environment. 

Through mental processes related to individual experience we recognize the ability to make us 
appropriately familiar with the environment that hosts us. Feeling comfortable in one place 
depends a lot on our relationship with light. Our perception of the visible allows us to 
understand and interact easily with what surrounds us by taking, even unknowingly, well- 
being through the pleasure of being part of that place and to appreciate its features without 
effort: the pleasure of pleasing and be welcome. 

If the objects had the possibility to speak and the rooms had the possibility to be walked, this 
would be certainly due to light. If this is accepted spontaneously, less spontaneously we 
accept that objects and environments have their own independent experience with respect to 
those who use them. Yet phenomena occur in them which do not depend on the will of the 
user, but rather, express conditions that suggest moods, purely emotional perceptions. 

The logic that the isometry of the form is to determine the functions or destinations of use is 
often a logic of perceptual suggestions, of relational relationships between light and shadow. 
Light not only illuminates, but characterizes shapes and spaces. 

Not only it allows perception, but it is precisely through it that emotional relationships are 
born. The objects, the spaces through the modulation of light, chromatic declination, tonal 
intensity, offer varied fruition dimensions. 


72 


Namely that if the physical space always remains the same in time, the luminous space can 
change and transform the perceptive conditions (by) changing the psychological relationship 
of things’ fruition in a cognitive or even suggestive way. 


Designing by light 

Designing light today also means refurbishing methodologies of approach to the discipline by 
adopting training paths that are able to interact in close partnership both to the design of 
physical parameters and to the project of intangible contents. 

To all this we add education to perception as a cultural structure. As Arnheim (1974) has 
already amply advocated, perception requires time, and in this it is necessary to educate 
oneself to be aware of it. 

According to Arnheim our perception, in a general sense, does not differ from what is 
necessary. That means that the concepts remain as generic as it is allowed by their application 
in understanding what one is observing. 

To perceive an object as immutable means to abstract it at the highest possible level of 
generality. In the physical world, modifications do exist. The control and design of this 
variability condition contributes greatly to the well-being of fruition to be more corresponding 
to what we actually see; otherwise everything would be immobile enough to make us argue 
that the variations either do not exist or do not matter, renouncing the emotional sensoriality. 
On this heightened awareness, accelerated by the evolution of technological possibilities, 
today new aspects of the project of light are developing. By now skilled in scientific mastery, 
the ligthing designer needs to tap into the sweet sensuality of light as a research tool for visual 
well-being and subsequently as a method of rationalizing perceptual cognition. 


Emotional Intelligence 

How the perceptual cognition of light has changed and how the need arose to relate it to 
emotional sensoriality can be highlighted by the development that the studies had on the 
influence that emotions exert on our perceptive abilities. 

According to the most recent studies, emotions contain information capable of enabling some 
sensorial capacities in order to read phenomena and influence our perceptions of well-being in 
a qualifying way. It is now clear that emotions characterize a very discriminating part of what 
it means to think, to make decisions and to solve problems. 

Yet until not long ago, emotions were still understood as an element of disturbance and 
fragility in cognitive processes. Only as from the end of the eighties in psychology there is a 
strong interest in emotions. 

The first signs of the interest of neuroscience to emotional phenomena were already triggered 
in the twenties by Edward Lee Thorndike, who proposed the concept of "Social Intelligence". 
To arrive at this codification, he paid attention to define the concept of halo effect. 

Indeed, he argues that a single positive or negative aspect of an individual can generate a 
positive or negative halo on other individuals. 

At the same time the studies of Robert Sternberg and Howard Gardner were in turn 
orientating themselves on other perspectives. Overcoming the classic "IQ", they began some 
researches based on models of Multiple Intelligences, which included the concept of 
"Personal Intelligence" 

In 1990, J. J. Mayer and P. Salovey finally brought order to the complex constellation of 
studies on the subject by definitively launching a real new branch of research. 

The contribution of Emotional Intelligence, however, did not have a big echo in the public 
right from the start and much less was influential on the culture of the project, which was still 
strongly linked to functionalist principles, despite the gain of the Post Modern. 

Instead, who contributed to its popularization and popularization was Daniel Goleman, who 
studied the works of Mayer and Salovey and attended their seminars: in 1995 he published the 
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famous best-selling book "Emotional Intelligence: what it is and why it counts more of the 
Io". 

Goleman, moreover, has led the research beyond the theorization of Mayer and Salovey, by 
adding many other connotations within his definition of emotional intelligence and often 
going beyond the scientific evidence of research on the topic. 

If, on one side, the volume of Goleman has had the merit of making Emotional Intelligence 
known to millions of people, on the other side it has generated lots of confusion in this field 
of studies. In fact, today there is no single definition of Emotional Intelligence. 

However, the studies on neurosciences conducted so far bring to our attention the evidence 
that the emotional centers of our brain are not hierarchically in a secondary position compared 
to the centers of what we could define reasoned cognition. But, on the contrary, these two 
dimensions communicate in a complex, harmonic & systemic manner which are strictly 
interconnected. 

From now on, the culture of the project cannot separate itself from this consideration and start 
from the emotional reactions in order to shape the spaces and define the shapes by using the 
new technological possibilities of lighting. 


The bright climate 

To design light, we are now in an advanced phase where we can easily understand that the 
choice of lighting instruments and their positioning is no longer enough, even if determined 
by the characteristics of the type of light. We are able to pay attention and reason on which 
bright climate we want and we can get. The thing we want is nourished by the regeneration of 
the culture of vision in reference to the new cultural skills of perceiving space. The way to get 
it is based on the capillary and experimental knowledge of the new technical skills to produce 
light. 

The Bright Climate is a component that those who practice the design of the environments 
consider taken for granted: One of the obvious events in which the project stumbles. In fact 
what is taken for granted is also inefficient. 

Having tools to understand what mood and state of mind can be suggested or take care of is 
not easy, considering the innumerable incident variables. Meanwhile, the subjectivity of the 
user changes anthropologically from one person to another, just like the induced variables of 
the time factor. Our knowledge of light is based on variability. Natural light is never equal to 
itself and independent from our will. This is our reference experience in perception. In the 
light that we design everything is instead fixed, determined, but above all voluntary. It 
depends on a choice of who designed it or who uses it. This may seem to be an advantage and 
until now in the culture of the project it has been like this; an excellent and ample benefit to 
shed light on request. But the potential offered by new technologies opens up completely 
different scenarios and conditions. More knowledge of both technical culture and perception 
culture is needed to focus on new choices and new formulations of lighting requirements. 

The bright climate today has many possibilities to be built, but needs to understand what kind 
of space to define based not only on the needs, but also the way to live it, to interpret it in a 
variable and transformable way. 

The simple act of "highlighting" is a design fact, based on choices, on responsibility and 
however spontaneous, rarely completely conscious. The act of sheding light today more than 
in the past needs information, knowledge, skills to give not only light, but the possibility to 
perceive our actions with agreeableness and acceptability. 


Photoluminescence 

The tools that the culture of light design has already refined converge largely towards the 
acquisition of the founding principle that “light can be declined as an emotional value”. The 
light passes from a necessary condition to see to a perceptive element that arouses moods. 
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"Kulana Nalu" by Pierandrei Associati 
Experimental set up realized for Lucedentro srl 


The light used as an expressive language can be translated into symbolic value. The cultural 
direction of perception greatly orients the suggestible capacities of communication. The hotter 
is the light the more it works as an attractor. It becomes "fire" to collect people around. 
Emotionally it generates a symbolic perception, an imaginary that supplies security and 
comfort. 

The fact that light is a symbolic emotion tells us that it is not possible to find the same path in 
the attribution of symbolic value both for “warm” and “cold" light. 

If the warm light associates an emotional imagery that welcomes and warms, for the cold light 
we are in a cultural evolutionary stage in which it becomes commonplace to associate the 
colors that go towards the blue at low temperatures. 

Today the symbolic value of this graduality of light is associated with new technologies. It 
combines with the most advanced performance of innovative materials. 

In this new perceptive perspective, photoluminescence is included. 

The signs of change can be appreciated, with various perceptive fluctuations, precisely in the 
ability of the new generations of photoluminescent pigments to be a vector of cultural 
orientation. 

To appreciate the considerable shift in the cognitive parameter we are subjecting to, we must 
first reaffirm a basic theoretical concept that has accelerated this ongoing process. There is no 
light but there is light in all its forms. It seems an axiomatic, purely conceptual postulate, but 
it has been the most significant stimulus that in recent decades has given awareness to the use 
of light as an emotional component. 

Says Vittorio Storaro: “We must continue to learn the art of seeing. The eye must seize ... the 
whispers of the visible”. 

The availability that today gives us the photoluminescence is to educate us to an emotional 
perception of the whisper of light. Doing little with the very imagination we need. The 
photoluminescence reinterprets the shadow light dichotomy, on/off, overcoming it by 
proposing a faint light, but useful to illuminate small spaces. 

In front of it the eye is purified from the overload to which artificial light has accustomed it 
and, as in a moonless night of stars, slowly rediscovers the complexity of the world and new 
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magical dimensions. Luminescence thus becomes a friendly, comfortable form useful for 
regenerating psychological and environmental energies. 


The chromatic pleasantness of light 

The common experience leads to understand that the objects are presented mostly with a very 
specific color. This leads us to believe that color is an exclusive and constant property of 
objects and surfaces of environments. Less spontaneous is the evaluation of what really 
happens to the color. It depends fundamentally on enlightenment and it is known through 
knowledge acquired by culture and certainly not by direct perception. 

Rather than understanding what color objects, surfaces, volumes & spaces are, we are urged 
to learn the information that color leads to acquiring. 

The new lighting technologies, the new lighting possibilities, considerably shift the interest 
from visual perception to visual comfort. 

It is a passage that is increasingly needed for awareness. 

The passage is almost epochal: the shift From the interest in the color of objects to the project 
of the color of light can be explained as a cognitive paradigm that is fundamental for the well- 
being of perception beyond perception itself. 

The growing awareness of how and how much certain aggregations of colors influence the 
mood and cognitive perception of a space can not sustain itself on the spontaneous notions of 
differentiation between warm colors and cold colors. 

Giving consequently the artificially cultural and simplistic meaning of excitement, 
aggressiveness, positivity to the so-called warm colors and instead calmness, privacy, 
relaxation, tranquility, negativity to the so-called cold colors the project methodologies are 
categorized rather than favoring their application in terms of benefits the potential that 
innovations allow. 

“I tried to express the terrible human passions through red and green”. Vincent Van Gogh 
was already aware of this categorization. 

If we want to understand the perception of color today, we must ask ourselves not only how it 
works, but also what are the ideas that men have made of it and how we intend to use it today. 
Surely among the most motivated applications there are processes that imply it to be effective 
in giving wellbeing. 

And it is the wellbeing that is subjected to a new exploration because not only is it not easy to 
define it, but it is amply demonstrated that its effectiveness depends strictly on the variants of 
its perception. 

The well-being as a guarantee of continuity of a certain state of health, of comfort, gives way 
to the increasingly variegated progress of the perception of the pleasantness of the 
phenomenon. Character completely subjective and closely linked to the social and cultural 
environment in which the observer lives and moves. Rather than being interested in wellness, 
perhaps it is necessary to consider the motive that leads us to perceive it: pleasure. 

It is through the pleasure we perceive well-being. 

Pleasure comes from emotions, it arouses emotions and variable effects. 

It allows to participate in a personal way to the mutability of things, of life. It allows us to 
escape from the compactness of certainties and the fixity of the perception of the world. It 
benefits the variety, the variability, the vastness of interpretation of what we see, perceive and 
which by nature is elusive. 

The pleasure of appreciating a color takes place before such a perception can manifest itself as 
well-being. 

The pleasantness thus becomes an attitude that is still little explored and enormously 
interesting, especially if considered in relation to the growing sophistication of visual 
languages and cultural complexity we live today the awareness that light is not only a 
perceptive feature, but a category we think about light with. 
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To give importance to pleasure, as a discriminating element of our perceptions, we need to 
start from the consideration that in our contemporaneity enlightenment takes on the value of a 
complex code of communication. 

To better understand it, we must distinguish our perception of color with respect to the source 
of illumination. 

At the root of our concepts of perception there is a misunderstanding that must be debunked 
and exists in the true color of things. 

But even his substitute must be debunked that is that to distinguish a true color you need to 
put it under the right light. 

And here is the first of the conventions to be overcome. Considering the sunlight more 
reliable than that of a light bulb is a conventional guarantee certificate and commonly used for 
instrumental acceptance for productivity purposes. 

We have already widely acquired by convention, not only in the practice of human narration, 
but especially in evaluations of industrial products, that the colors are estimated by placing 
them under the standard light of 6500 Kelvin degrees and it resemble the temperature of the 
sunlight of a generic noon. 

The light, therefore, takes on the role of leading architect in giving ways to describe and 
furnish the space. 

Today we are creating a global approach to color space. A perspective set to be considered 
together as a complex play of lights and reflecting surfaces that can be regulated by the 
infinite possibilities of light modulation. 


The experience of the new immaterial dimension of light 

We are now widely used to see environments that change color only with the temporary effect 
of artificial light. So, we are culturally predisposed to accept and perceive, as a value, 
intangible light effects that can change the visual perception of the place. 

Since the 1960s, pop art has been able to accept advertising light as the most advanced 
expression of the lighting technology that modernity could have. 

Las Vegas and Times Square have spread the culture of diffused light. 

The sixties were the years of fluorescent neon inaugurating "the era of the naked bulb", or the 
luminaire in sight. 

The jump happened because there was no need to hide the light source anymore. Everything 
is left to the eye. The lamp becomes no longer just a source, but a legitimate medium in itself 
that characterizes the colors of the spaces. 

The result is the diffusion of a fluid, magical, undulating weightless perception. 

F. Wright writes in the “Testament” - “Glass pipes superimposed like bricks of a wall make 
up the luminous surfaces” - we are in the age of “diaphanous” light: light that envelops, light 
as much as possible. 

With the advent of digital control of artificial light, we can define a new immaterial 
dimension. To the conventional dimensions is added the emotion produced by the varied 
applications of lighting technologies. The point control of each lighting effect can be 
reactively managed with the environment, in direct correspondence to the needs of the 
moment and to vary according to the perceptive conditions of the users. 

Light from an immaterial dimension becomes a participatory immaterial dimension: it 
produces sensations and stimulates immersive perception. A new dimension that needs more 
awareness in the organized scandery of the fruition of space. The immaterial interconnectivity 
of objects, which is now expanding, has the concreteness of existing if it finds in the lighting 
apparatus the dimension in which to express the usefulness of its own abilities. 

The light is totally involved in creating both the technical functionality of the connectivity 
between the objects, and the LI-FI is already a physiological reality, both in creating the 
environmental conditions. By doing this, the new invisible prodigies of technology can be 
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perceived by humans. 
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Project by Viabizzuno srl 


In this, now an advanced performance scenario, the lighting project overcomes, with ever 
more sophisticated implementation tools, the commitment of providing a correct contribution 
in lumens to the environments in order to quickly move to a new methodological application 


phase. 


Moving the ligthing design towards a strongly cultural trajectory: educating to 


wellbeing by educating to the perception of light. 
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Robot Surgery for Total Knee 
Arthroplasty 


Prof. Michele D’Arienzo! 
Prof. Lawrence Camarda! 
Dott. Antonio D’Arienzo? 


"Department of Orthopaedic Surgery, University of Palermo, Palermo, Italy 
"Department of Orthopaedic Surgery, University of Pisa, Pisa, Italy 


In order to reduce complications and to improve patient satisfaction after total 
knee arthroplasty (TKA), in recent years many technological advances have been 
introduced. Most of them aim to better place the femoral and tibial components, allowing 
a reduction of the risk of uncorrected prosthesis placement. In fact, improved alignment 
has been shown to increase implant survival and decrease revision surgery. In this 
scenario, robotic surgery has become an increasingly popular tool for orthopaedic 
surgeons. Robotic surgery has the potential to enhancing the surgeon’s ability to 
generate reproducible techniques through an individualized surgical approach, optimising 
the soft-tissue balancing and restoring the normal joint kinematics. The most common 
robots commercially available work as a semi-active systems providing passive haptic 
restraints for surgical resection. In this condition, the surgeon can perform bone resection 
under the supervision of the robot, confining the cutting bone to only the planned level of 


resection. Different alerts trigger (vibration, beeping, colour change on the computer 
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screen) provide feedback to the surgeon as the defined resection parameters are 
approached. Surgical steps include 1) creation of a patient-specific knee model 2) 
registration of the knee model and plan the bone resections 3) the use of the robotic to 
make bone cuts and carry out the preoperative patient’s plan. During surgery, the 
position of the burr is evaluated by the robot and when it approaches the border of the 
planned resection the computer system will slow the speed of the burr or retract the burr 
into the hand piece, effectively decreasing the potential for over resection of the bone. 
During this phase, surgeon could make adjustments to the patient’s plan in order to 
obtain the most accurate placement of the knee prosthesis. Surgical procedures are 


demonstrated such as clinical results of robotics surgery. 
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“We all die, but looking” is the title of the autobiography of Alberto Garcia Alix, one of the 
most representative photographers of 2nd half of 20th Century in Europe. It is impossible 
summarize better, in a few words, the relevance of the natural capability of human being of 
catching everything that surrounds him. The human flair of converting images into memories 
contribute on building what we were, what we are and surely what we will be. Since last 
1895 the memories of history of human being has being helped by the possibility of capturing 
and saving this part of our evolution into photographs and movies. These physical supports 
have saved main part of our global cultural, social and political history. But, looking into 
each individual, also gave the possibility of recording part of our family heritage. 
Photographs and movies have helped to preserve the cultural material that is faithful witness 
of socio-cultural European evolution in the recent era. It encompasses not only the possibility 
to understand the development of new arts such as cinema, photography or graphic arts but 
also the preservation of the socio-cultural memories of citizens in major museums, local 
museums (real testimony of the history of cities and their people) located in many towns 
worldwide & in a huge number of private collections. A huge percentage of the recent 
European cultural heritage (CH) can be found in movies, photographs, posters and slides 
produced between 1895 to nowadays were made using cellulose derivatives. More than 75 
years of visual and audio memories are up to now in serious danger to be lost due to the 


natural instability cellulose acetate (CA) and cellulose nitrate (CN) materials. The 
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worldwide estimation of such holdings within professional film archives is around 18 
Mio of film reels on cellulose acetate, whereof ca. 5% are in a critical stage or showing 
signs of vinegar syndrome (survey carried out by the OEAW-PhA in 2012, performed with 
20 institutions, extrapolated to 125 members of FIAF (Fédération Internationale des Archives 
du Film). Figure 1.1.a shows an example of such film. There have had other technical 
approaches to solve this real problem that follow the line of replication and copy the original 
ones in modern digital supports (for example PIQLBox©- www.piql.com/). But, they do not 
give the possibility of a real preservation of the original ones. In the actual technology age, 
our duty, with the people that preceded us, pass through help on the preservation of these 
parts of original memories that built our past history and will help on understanding our 
future. Under the technical point of view, conservators consider two approaches when 
planning treatments to extend the useful lifetime of cultural materials: preventive or passive 
and active or interventive. However in the case of cellulose derivatives and other components 
of the movie or photography, once initiated, degradation cannot be prevented, reversed or 
stopped, but only inhibited or slowed. Inhibitive conservation of cellulose derivatives can 
either involve the removal or reduction of factors causing degradation including light, oxygen, 
acids, fungus and relative humidity, or of any breakdown products which accelerate 
degradation, as well as cost-sensitive processes such as freeze. The cellulose derivative 
degradations is autocatalytic increasing constantly the level of degradation with the time. 

NEMOSINE aims to improve the traditional storage solutions, such as freeze storage (below 
5°C), by developing an innovative smart package with the main goal of energy saving and 
extent conservation time of cultural objects based on cellulose derivatives. Beyond the 
state of the art NEMOSINE plans to develop the following modular and integrated products: 
i)High O2 barrier and Active packaging using non-odour additives, ii)Active acid adsorbers 
based on functionalized Metal Organic Framework (MOFs) integrated in low density and 
porous structures, ili)Gas detection sensors based on nanotechnology to monitoring AA, 
water, O2 & NO, iv) Multi-scale modelling to correlate degradation & sensors signals for 
maintenance prediction and integrate all these technologies in v)Packaging with modular 
design to fulfil the technical & economical requirements of the different CH made by 
cellulose derivatives and vi) Curative packages containing controlled release of natural 
antifungal additives. The main targets of NEMOSINE project are cellulose derivatives, from 


photographic, movies and audio substrates. 
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INNOVATIVE PACKAGING SOLUTIONS FOR STORAGE AND 
CONSERVATION OF 20TH CENTURY CULTURAL HERITAGE OF 
ARTEFACTS BASED ON CELLULOSE DERIVATIVE 


A huge percentage of the recent European cultural heritage (CH) can be found in movies, 
photos and posters produced between 1895 to nowadays were made using cellulose 
derivatives. More than 75 years of visual and audio memories are up to now in serious 
danger to be lost due to the natural instability cellulose acetate (CA) and cellulose nitrate 


(CN) materials. 


NEMOSINE aims to improve the traditional 
storage solutions, such as the usual freeze 
storage (below 5°C) that involves high energy 
consumption and special facilities, by 
developing an innovative smart package with 
the main goal of energy saving and extent 
conservation time of cultural objects based on 

cellulose derivatives. 


NEMOSINE smart package concept 


Sensoric devices High durability Multitayer 
AA, NO, external plastic layer package 


AA Absorbers (MOF) “ Open cell foam, 
Antifunger i jers matt or 


CO, barrier na 
plastic layer Moisture Absorbers cellulose paper matt 


Beyond the state of the art NEMOSINE 
plans to develop the following modular 
and integrated products: 

j) Active packaging using non-odour 
additives. 

li) Active acid adsorbers based on 
functionalized Metal Organic 
Framework (MOFs) integrated in 
low density and porous structures. 

iii) Gas detection sensors based on 
nanotechnology to monitoring AA, 
water, NO. 

iv) Multi-scale modelling to correlate 
degradation & sensors signals for 
maintenance prediction. 

v) And integrate all these 
technologies in on smart packa 
with modular design to fulfi 
technical & econog 
requirements of the diffe 
made by cellulose deriva 

vi) Curative packages 
controlled relea 
antifungal additi 


The main targets of NEMOSINE project are cellulose derivati 


audio substrates. 
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munity is related to digitisation and digital 
which is stl very complex, costly and time consuming. As 


WP) WET Design ofa amor package 
tr longt re Gi ponn, poren, 
photagapte anciuiiciar, noaga 

roars iergerse. z 


The main beneficiaries are the audio-visual collections can be grouped as follows: 


1) (Partly) autonomous audio-visual 2) Materials privately owned by 3) Private owners that 
research archives;memory the researchers, collectors or keep own old family 
institutions, such as libraries, big audio-visual buff that created photographs (and/or 
archives and museums; audio and video documents their negatives), 
departments of research or cultural for their research purposes. movies or audio. 
institutions, local museums, oral 
history societies, etc 


CONSORTIUM 
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Abstract - Museums are experiencing a shift in their roles within society; a shift in revolutionizing 
the way museum guests access objects and participate in museums’ collections. Contemporary 
museum guests desire more active immersion in, and interaction with, museum galleries, desiring 
direct contact with cultural and historical artifacts, and forums to discuss their experiences, aided 
by technology. To meet this need, The Museum of Texas Tech University is leveraging 3D Mixed 
Reality (MR) technology (zSpace®) to tap into these societal shifts; providing new ways for museum 
guests to conceptualize, communicate, and connect by replicating historical and cultural memories 
of Native American objects. 


INTRODUCTION 


Traditionally, museums have regarded themselves as temples of knowledge; physical spaces 
to house cultural and historic objects memorializing human culture across time and space. And 
thus, historically, these spaces were designed as formalized places of learning, with prescriptive 
architectural designs and specific language used for signage in exhibits. For decades, guest 
experiences at museums were defined by viewing objects under bright lights and thick glass; a 
‘look but do not touch’ experience with the protection of collection objects. Yet, with dwindling 
museum attendance, especially among young adults [1], museums have begun to examine their 
approach towards exhibit design and purpose, reconceptualizing how visitors may have 
evocative and educative experiences with museum collections in new and interesting ways. 


Fostering Interactive Exhibits: A Paradigm Shift 


Museum professionals now seek to reimagine traditional perspectives of what constitutes the 
visitor experience, broadening object experiences (focus on the artifact) into social, cognitive, 
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and introspective experiences; museums of are interested in how guests interact with each other, 
learn, and feel from their museum experiences, respectively [2]. Specifically, museums are now 
interested in exhibits that contain a combination of opportunities for guests to both acquire and 
share knowledge, and this hopefully results in the creation of emotion and reflection on the part 
of the visitor. To this end, research suggests that exhibit design should leverage aspects of free 
choice learning that motivate and empower visitors to learn about new historical and cultural 
narratives [3]. 


Theoretical Framework: Learning in Free Choice (Museum) Environments 


As this paradigm shift progresses, museums are in need of devising plans to create 
educational and collaborative spaces for their public to participate in informal or free-choice 
learning activities [4]. To understand this body of informal education literature in which 
museums exist, we must first examine determined modes of learning, combining theoretical 
perceptions and modern understandings of the way today’s societies access information. From 
decades of research, Lynn D. Dierking described ten learning generalizations that are 
influencing factors when understanding the context of the human learning process [5] in 
informal, free choice learning environments. These ten processes of learning generalizations 
describe that one’s perception (1), motivation (9) and memory (2), informed by prior knowledge, 
beliefs, and experiences (7) and development (5) are central to one’s learning. And, this learning 
occurs through nonlinear (6), active (3), cognitive and metacognitive processes (4), within 
physical and social contexts (8) that are unique to the individual (10). Quantifying the complex 
factors that influence the general processes of learning has the potential to expand our traditional 
methods of exhibitions methods of exhibit design, and to change our perspectives about ways to 
reach broader audiences. Therefore, in using these generalizations, museums can further their 
consideration of what constitutes methods of exhibit design and content, which can help to create 
a more educational and emotionally meaningful experience for museum guests. 


Conceptual Framework: How Technology Can Enhance Learning (and 
Museum Goals) in Free Choice (Museum) Environments 


Technological developments continue to shape and change the way we interact with 
everything we encounter, and this has been evident in the use of virtual, augmented, and (when 
presented together) mixed reality technologies. These include realistic and interactive 3-D 
images set within virtual environments, which stimulate, motivate, and enhance learners’ 
understanding of events or phenomena, especially when traditional (hands-on) experiences are 
prohibited or difficult [6]. Such technologies have been found to aid museum guests’ knowledge 
of artifacts as well as facilitating guest excitement and motivation [7]. Yet, most prior research 
has focused on replication of objects into various environments, by either placing the actual 
object in a sterilized setting (e.g. under glass) or creating a replica of the actual object and 
placing it into an approachable setting (e.g. in an exhibit where people may touch or see the 
object in a context). In either scenario, these actions still divorce the object from its cultural, 
social, and historical context. Yet, this should be important to museums, as traditionally the most 
important element within the presentation of these objects is the authenticity of these objects, 
namely their cultural affiliations and associated historical information. The general presentation 
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under glass is waning; museum professionals are reexamining what it means to have an authentic 
experience and are applying new conceptions of the term authenticity to expand the methods of 
public exhibitions. In a 2010 publication titled Fabricating Authenticity: Modeling a Whale at 
the American Museum of Natural History, Rossi explained the qualifying features of authenticity 
when addressing using replicas within museums stating that, “Through these records, one finds 
that practices used by exhibitors to replicate whales were intimately tied up with questions of 
truthfulness, accuracy, authority, and moral and aesthetic worth—qualities that I will sum up 
under the rubric of authenticity.” [8] (p. 340). These defining qualities of authenticity allow for 
those features to be applied to museum objects, regardless of their creation story and suggest that 
all objects that are modeled through accuracy, and that are morally and aesthetically truthful, 
could only then be considered authentic. Prior to this work, for decades—perhaps centuries—, 
an authentic experience was defined by visiting a museum and viewing objects under bright 
lights and thick glass. This ‘look but do not touch’ experience privileged the protection of the 
collection, with latter consideration to the guest. However more recently, museums have started 
to examine their approaches of conceptualizing exhibit design, conceptualizing how visitors can 
have meaningful experiences with the collection in new and interesting ways. Rossi clarified this 
shift in focus, stating that “although there was no decisive moment of this epistemic shift-old 
techniques for insuring an exhibit’s moral, scientific, and aesthetic worth continued to find use 
alongside more novel tactics—exhibitors increasingly tended to view scientific authenticity 
governed not by objects, but by the experiences of viewers as participants.” [8] (p.359). This 
suggests that the incorporation of emergent technologies (like virtual, augmented and mixed 
reality that seek to replicate the real world) within museum exhibits could now allow for 
museum professionals to redefine the notion of what constitutes an authentic experience within a 
museum setting. Museum professionals are now seeking to reimagine our traditional 
perspectives of what constitutes a visitor experience, making efforts towards curating new types 
of guest experiences within traditional exhibit spaces [9]. Meaning that exhibits could provide a 
combination of opportunities for gaining knowledge, combined with opportunities that induce 
important features of generalized learning, such as, evoking emotion, developing memories, and 
promoting self-reflection. When challenged with the idea of pairing an emotional response with a 
visitor’s museum experience, museum professionals could provide a setting within which visitors 
are able to create their own understanding of authenticity, in turn allowing museum staff to 
redefine traditional notions of museum exhibits. One way that museums can be successful with 
shifting the idea of traditional exhibits is to focus on transforming the feel (italicize) of their 
exhibits; shifting from traditional, formalized “do not touch” exhibits, to exhibits that provide 
interactive elements, a museum can provide a visitor with a more personalized, collaborative, 
and emotional learning experience. Hence, it is paramount that exhibits encourage motivational 
and free choice learning for guests, so they may feel empowered in their learning about (new) 
historical and cultural narratives. 


The Museum Collection: Southwest Indian Art Gallery 


The Dierking generalizations [5] and ideas of robust virtual experiences [6, 7] fostered the 
framework of creating dynamic and educational visitor experiences, to be applied in a real-world 
museum setting at Museum of Texas Tech University; the idea for creating robust, virtual 
experiences was specifically applied to the William C. and Evelyn M. Davies Gallery of 
Southwest Indian Art Gallery, “Beyond Expressions in Clay” [10]. This exhibit includes objects 
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(e.g. pottery, textiles, ceremonial and trade vessels) from 20 different Native American tribes in 
the Southwestern United States, spanning from the eleventh century to the twentieth century 
A.D. Many of these objects are far too fragile for display and are kept in the archives; those 
artifacts that are on display are under standard, museum protective glass. Many visitors to the 
museum and gallery are not Native American, and require museum-based scaffolding so they 
may be able to connect through their perception, motivation, and social interaction to name a few 
(related to Dierking’s ten generalizations [5]), to foster robust learning and appreciation of the 
collection. Since museum guests hold little (if any) prior knowledge of, or experiences with, 
Native American culture and beliefs, it was important to try to create some kind of geographical 
and cultural context for the objects included in the project. By using original photographs from 
actual locations that are associated with selected objects, the goal is to foster a more meaningful, 
personal experience with objects in the collection. Mixed reality technologies were employed to 
replicate objects’ authenticity, or the inherently situated nature of the social, historical and 
cultural relevance of the collection’s artifacts. 


Mixed Reality Technology: The zSpace® Platform 


The zSpace® VR hardware is desktop-based that uses stereoscopic images to produce 3-D 
images. The technology uses elements of both augmented and virtual reality (within a modified 
desktop computer) to facilitate mixed reality, immersive and interactive experiences that 
promotes user-driven active learning. Figure 1 demonstrates the components of the zSpace” 
system. 


Figure 1. The zSpace® 200 display. 


The zSpace® unit consists of: a central processing unit (CPU); a 24-inch-high definition liquid 
crystal (1080p, 120Hz) 3-D stereoscopic display screen complete with built-in tracking sensors 
to track the viewing angle of the user; a 3-button stylus with integrated haptic technology and 
infrared LEDs for manipulating interactions within the virtual reality space; and a set of 
polarized eye-glasses with reflective sensors to track head and body movement in real-time. The 
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system’s monitor with specialized 3D glasses creates the perception of depth on the z-axis, 
making the objects on the screen stereoscopic, creating an immersive environment with apparent 
realism. The glasses synchronize with the zSpace™ head tracking technology to sense how the 
user is moving their head and which direction they are focusing their view in real time to 
enhance the sense of user interaction. zSpace® also incorporates an attached specialized stylus 
pen that allows the user to interact with an object and manipulate it to get a 360-degree view. 
The additional zoom in/out capability allows for examination of the objects as they ‘hover’ in 
front of a user. This hardware, coupled with 3-D VR software applications, allows the viewer to 
see detailed, simulated images, that appear both within and out of the screen, and allows 
manipulation (e.g. rotated, zoomed, dissected, etc.) of these 3-D images with the stylus. zSpace® 
was chosen based on unique affordances among available mixed reality technologies. First, 
compared to other forms of augmented and virtual reality systems, this technology is relatively 
inexpensive to acquire. At $3000 to $5000 USD per unit, the zSpace® system allows for smaller 
museums to have much more affordable access to incorporate emergent technology within their 
institutions. Second, zSpace® allows for a dynamic and fluid informal learning environment. For 
example, one or two zSpace® systems could be incorporated as enhancements within 
traditionally styled museum exhibits, or, multiple zSpace” systems can also be used in a 
‘discovery lab,’ creating a space for more guided interaction. Lesson plans can be continuously 
updated and customized based on specific learning objectives, audience, or age groups. Third, 
zSpace® is user friendly, which has been leveraged by prior research studies due to user 
familiarity of the computer style monitor, stylus pen, and glasses, hardware that is within the 
general public’s lives and range of experiences [11]. Therefore, making the user interaction 
process less intimidating than similar emergent technologies. Fourth, the zSpace” employs a 
stereoscopic display that provides peer-to-peer interaction, unlike other forms of virtual reality 
hardware for single-users only, [12], and this foster rich conversation and interaction among 
museum guests while they are engaged with the technology. Last, the zSpace® lesson plan 
development process can be easily learned. A potential developer (with no formal training) could 
successfully scan and import artifacts to create custom lesson plans for objects within their own 
collection. 


Integrating zSpace® into a Museum Collection 


The goal of this ongoing project is to provide museum guests a better understanding of 
collection objects through the use of immersive and interactive 3D experiences via the zSpace” 
platform. Aside from the obvious entertainment value, the opportunities of enhancing the guests’ 
educational and affective experiences with collection objects are immense. Informed by the 
Dierking theoretical framework of learning [5], objects from the Davies collection were scanned 
into fully-rendered 3D images (using an iPad with the 3D scanning application itSeez3D) and 
then imported into the zSpace® system software (zStudio"). Building upon the provenance and 
extant research of the objects, culturally representative music and appropriate scenery were 
added to the selected object’s virtual environment (as seen in Figure 2). 
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Figure 2. The zSpace 200 display of a Davies’ Gallery Object. 


This included 3-D models of the collection’s objects, replete with Native American music, 
environments, pronunciations, as well as guided activities and questions to help guide the user 
experience (as seen in Figure 3). 


Figure 3. A user interacting with an object in the zSpace® 200 mixed reality environment. 
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Now that objects from the Davies’ collection have been scanned and loaded, and, lessons 
developed on their histories, a zSpace” system is now deployed in the Davies’ collection as of 
fall 2018. In 2019, a full research study will explore guests’ experiences within the exhibit. 


Conclusion 


Despite many museums’ lack of funding for innovation, it is crucial to keep in mind that 
despite the evident academic and educational benefits for the public, museums are now in direct 
competition with other forms of leisure entertainment [13]. We must shape our museums by 
keeping this fact in mind with the understanding that if we want our institutions to remain 
relevant to the public, we must reinvent an authentic guest experience. The ultimate goal of the 
zSpace® intervention is not to act as a replacement for the traditional method of ‘look, read, but 
do not touch’ that is found within traditional museum exhibits, but rather as an enhancement to 
exhibits and objects. It is our purpose to provide our visitors with immersive and interactive 
virtual experiences with 3D, digital representations of collection objects, so that visitors are able 
to have novel, more authentic experiences with the artifacts, like never before. With our 
research, we anticipate zSpace® will help fulfil our Dierking-informed mission of learning [5] 
where guests may actively learn, discuss, explore and create, fostering memorable museum 
experiences and long-lasting learning. 
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This paper describes an approach of employing a dual heritage imaging process in an attempt to 
overcome the inherent challenges of inscription analysis and surface texture information in several 
different situations. 

In the cases presented, we have employed a hybrid approach, cross-referencing traditional 
archaeological research with close-range 3D Structure from Motion (SFM) imaging and Reflectance 
Transformation Imaging (RTD. 

As will be demonstrated, it is our opinion that in the circumstances such as these, there is not one 
dominant technology, but rather, a combination of complementing strengths, which can aid in unlocking 
the secrets of the engravings and inscriptions. 


Jerusalem. Church. St Helena. Crosses. RTI. SEM. Milestones. Olmec. Seals. Tombstones. Museum 


1. INTRODUCTION 


In the fields of Heritage conservation, in restoration, analysis, measurement, research and 
presentation, a vast number of both invasive and (increasingly) non-invasive techniques are used to 
extend the range of our senses. Some peer and probe beneath the surface of the object, in search of 
visual clues, whether by multispectral methods, by radar, by LIDAR, or by magnetic resonance. 
Others, such as XRF and XRD looks at the elemental or chemical characteristics of substances. 
Microscopy extends the range of sizes accessible to our human visual system, while drone, aerial and 
satellite photography provide access to hitherto hidden vantage points and offer an overview of large 
surfaces. 


Yet, much of the information we wish to view lies in plain sight is neither too big nor too small. Lurks 
not beneath the surface nor hovers above it. Does not extend beyond our visible spectrum and needs 
no chemical detection. Still, it remains hidden from our sight, due primarily to that one very source 
without which we would be lost: Light. 


Light reflects off, refracts from, or is transmitted through surfaces and reaches our visual receptors. 
However, beyond the effects of intensity and wavelength, it is the angle from which it reflects which 
determines our ability to interpret information. We are talking here of surface texture. The texture is 
viewable by the way the light reacts with it, the quality of light and especially the angle of light hitting 
the surface. Inherent contrast or lack of it in the subject matter will affect our ability to discern detail. 
Yet it most cases the characteristic of texture recognition will be a factor of elevation and depression 
relative to the light source, while the angle of the light raking off it will determine the intensity of 
highlights and shadows, which give the surface its depth dimension. 


However, in many cases, we are not the masters of the light source and are not in control of its 


interaction with the subject’s surface. A rock carving may be illuminated by a dull overcast sky, 
yielding it virtually unreadable, while a highly reflective surface may be subject to direct harsh 
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sunlight. Moreover, the nature, the intensity, and angle of light may be highly useful to one part of the 
subject, yet entirely unacceptable to another part of it. Curved surfaces reflect the light in different 
directions and thus by that very fact the optimal direction of light for viewing one part of the surface 
will be the wrong one for another. 


This paper chooses to concentrate on two low cost and readily available imaging techniques designed 
to overcome these issues: Reflectance Transformation Imaging (RTD and Structure from Motion 
(SFM). Both techniques have been around for some time and are reasonably well known and 
documented. However, their applications have traditionally been somewhat different and do not 
usually aim for the same purpose. 

Here, I wish to argue and demonstrate that it is the combined virtues of the two techniques that, when 
used together may offer an effective and affordable solution to high-quality surface detection. 


2. REFLECTANCE TRANSFORMATION IMAGING 
2.1 Theory 


Reflectance Transformation Imaging (RTI) is a unique digital photographic technique aimed at 
enhancing the surface detail of objects through an algorithmic rendering of multiple registered digital 
images of the object, shot with controlled, varying, yet known light positions. Termed initially 
Polynomial Texture Mapping (PTM), the method was invented by Tom Malzbender at the Hewlett 
Packard Labs. The seminal paper describing the method was published in 2001 [Malzbender, Gelb, 
Wolters, 2001]. Today the technology continues to be developed and promoted by a dedicated 
international team, supported by the Cultural Heritage Imaging Corporation (CHI). 

The power and beauty of RTI lie in its ability to enhance the surface-detail of a wide variety of 
materials by changing the apparent direction of light falling upon it. This 'virtual torch' [MacDonald, 
2011] is done after the photographic session has been completed, and at the leisure of the researcher in 
the comfort of the lab. During photography, the light sources are positioned at a constant radius from 
the subject and surround it at incremental angles, forming a dome or hemisphere of light positions. 
The user may also zoom in and out, change sharpness, contrast and other light and surface properties, 
through a series of real-time filters, thus often revealing surface details not visible to the naked eye 
under normal viewing conditions (See Fig. 1). 


2.2 Practice 


As the real-life cases here demonstrate, the RTI technology is very flexible, relatively easy to 
master and yields remarkable results. Yet, though relatively cheap and straightforward, it is 
nevertheless time-consuming and demands total accuracy in execution. These two demands, while 
acceptable in a controlled studio environment are in some cases challenging to achieve, due to 
environmental or time constraints. Furthermore, the fixed location of both camera and object limit the 
viewable area of the surface and the information it may yield. 


3. STRUCTURE FROM MOTION 
3.1 Theory 


Structure from motion (SFM) is a photogrammetric range imaging technique for estimating three- 
dimensional structures from two-dimensional image sequences that may be coupled with local motion 
signals (see: Wikipedia). It is based on the principle that while a single photograph can only yield 
two-dimensional coordinates, height and width (X-Y), two overlapping images of the same scene, 
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taken slightly apart from each other (a stereo pair) can allow the third dimension (Z) to be calculated 
through the process of triangulation. 

A photogrammetric model combines the elements of shape, texture, and lighting of the object and 
stores them as a combined vector and raster file, thereby providing the combined advantages of 
mathematical accuracy and flexibility, along with photographic exactitude of colour and texture. This 
process lends itself especially to 3D reproduction and representation of existing real objects and 
surfaces, rather than the creation process of new objects, based on copying or imagination 

Recent years have seen a significant advancement in this field, due both to improvements in 
algorithms and software, as well as the proliferation of high-quality DSLR cameras and high-end 
phone cameras. Consequently, today, there is a growing argument that for many purposes, 3D 
digitisation by SFM can equal and often surpass laser and structured-light imaging [Skarlatos, 
Kiparissi, 2012]. Thus, photogrammetry-based 3D imaging has moved from the status of low cost / 
low quality to that which challenges the quality of mid to high-end scanners in specific fields 
[Cultural Heritage Imaging, 2015]. 


3.2 Practice 


SFM is sometimes an unpredictable process. Some objects respond better, and some yield poor 
results. Much depends on the nature of the surface captured, the quality of the light sources and of 
course the quantity and quality of the photography. Sharpness and even lighting are of great 
importance, and though this may sound trivial, it is often challenging to achieve. In the cases 
presented here, environmental and temporal shooting conditions played an essential part in the level of 
success. 

As opposed to RTI, where there is just one main software approach to processing the images (CHI 
RTI Builder), SFM offers many software solutions of varying complexity and price. In the cases 
presented here, all objects were processed in at least 3-4 software packages. These included: 

e Agisoft Photoscan 

e Autodesk Recap 360 

* 3D SOM 

e 3D Flow Zephyr Aerial 

e Reality Capture 
Each software package has its advantages and disadvantages regarding speed, accuracy, flexibility to 
overcome shooting faults, hardware demands and so forth. Subsequently, it transpired that the best 
results were achieved overall with Agisoft Photoscan and 3D Flow Zephyr Aerial. 
Due to the subsequent file size of the photogrammetric models and the need for the researchers to 
access the models with ease and furthermore to annotate them with their findings, it was decided to 
upload the models to the Sketchfab web platform. This excellent site allows for post-processing, 
lighting, Meta tagging and annotating the models. 


4. TEST CASES 


For this practical research, we have concentrated on several test cases, which highlight the various 
complexities in surface information detection and the benefits of the combined RTI / SFM approach. 
These include: 

° The Saint Helena Chapel Crosses in the Holy Sepulchre, Jerusalem 
e Roman Milestones along the Incense Route 

e Jewish Tombstones in Germany 

e Cylinder Seals 
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4.1 The Saint Helena Crosses 


This case describes an attempt to provide insight into the riddle surrounding the thousands of 
crosses, symbols, and texts, inscribed on the walls and behind the altars of the Chapel of Saint Helena, 
within the Church of the Holy Sepulchre in Jerusalem, ascribed to the Crusaders of the 12" and 13th 
centuries [Pringle, 1993]. 

Working with the Israel Antiquities Authority in collaboration with the custodians of the Church, both 
RTI and SFM were employed, in a concentrated attempt to document as much empirical data as 
possible during a short period of 48 hours when renovations to the chapel opened up a small window 
of opportunity for access. 


Figure 1: Detail of stone (6X10 cm). RTI with Specular Enhancement & SFM Radiance Scaling (left). 
3D plot of a cross (right) 


RTI alone was not sufficient for this purpose. While the quality achieved on some stones was 
remarkable (See Fig.3), the sheer volume of tiles made this lengthy process impractical. Furthermore, 
the very short shooting distance between the back of the altars and the walls, the position of some tiles 
near the floor or high up, in niches or other inaccessible locations made it impractical to position 
tripods or move the light source in the required hemispherical direction. Lighting conditions and 
extremely thick stone dust (due to renovation work taking place) further hampered the process. SFM 
was in these cases a more convenient solution where the very close shooting distance available was an 
advantage. Nevertheless, those tiles which were accessible to the camera and lights yielded 
remarkable detail, both via RTI and SFM. Analysis of the chiselled incisions on the stone reveals 
regular V-profile grooves (See Fig. 1, left). The angles were determined by using a photometric stereo 
(‘shape from shading’) image processing technique to determine the surface normal vectors at each 
pixel position (See Fig. 1, right). 


4.2 Roman Milestones 


Last winter, a lost section of the ancient Incense Route (see: Wikipedia), stretching seven 
kilometres, was discovered in the barren heart of Israel's Negev desert. Along with this stretch, several 
Roman-era milestones were discovered. Two thousand years of harsh conditions had taken their toll 
and as a report in the local newspaper stated: “One of the Roman milestones still bears all-but- 
indecipherable writing, which seems to be in Latin or Greek. 

Under the auspices of the Zsrael Nature and Parks Reserve along with experts from the Israel Museum 
and the Tel Aviv University, detailed imaging of the milestones was undertaken. 

From the outset, it was clear that conventional photography was not going to assist much in 
recovering the faded letters. Thus, RTI was deemed the hopeful solution. Indeed, this was in some 
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cases so, enabling the confirmation or rejection of long-lost messages from the past, including the 
name of the Roman Emperor Septimius Severus (145-211 AD) (See Fig. 2, right). 

However, Roman Milestones are cylindrical and therein lies an inherent limitation of RTI, which 
works primarily on two-dimensional surfaces. Texts which stretch along the cylindrical surface cannot 
be captured in full. This is where close range SFM came into its own, giving the researchers full 
access at any angle, level of zoom and light direction. Furthermore, readily available 3d viewing 
software such as Meshlab, offers both interactive relighting and filters, such as Phong shading or 
Radiance Scaling, which aided greatly in accentuating the texts on the surfaces (See Fig. 2, left). 


Figure 2: Milestone: SFM and detail rendered in Lambertian Radiance Scaling (left). Detail bearing 
the name of the Emperor Septimius Severus. RTI unsharp & specular (right) 


4.3 Jewish Tombstones — Germany 


As part of an ongoing conservation project, Jewish tombstones in several small German 
communities were documented (See: Unfolding Communities). These communities all perished 
during the Holocaust, and until recently, their cemeteries too fell into disarray or worse still suffered 
vandalism. The conservation effort involved teams of young students from the local German schools 
alongside students from Israel and small teams from the Institute for the Preservation of Cultural 
Heritage at Yale University and the Hadassah Academic College Jerusalem. 

All the tombstones were photographed, and their data uploaded online. However, many of the stones 
have succumbed to the ravages of both man and climate, making their reading all but impossible. 
These stones were earmarked for special treatment, namely, RTI and in some cases SFM. 

The RTI documentation proved invaluable in its ability to bring out the faded lettering on the stones 
and even to restore long lost names, through which the story of the deceased was retraced and 
documented (See Fig. 3, left) In other cases, however, the conservation report went beyond the 
lettering and related to the general state of the stone, its cracks and its subsidence. In cases such as 
these, SFM provided a better all-around representation, complementing the RTI (See Fig.3, right). 
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4.4 Cylinder Seals 


Cylinder seals are small, carved stone cylinders that were used to make an impression in wet clay. 
When rolled out the seal left an impression that could prove ownership or identity. 
Cylinder Seals are in most cases extremely difficult to view and decipher due to their size, material, 
and shape. Their sizes can vary between a mere 1 cm to possible 3-4 cm at most. Thus, unaided and 
non-invasive viewing is a challenge from the outset. Secondly, the hard, often shiny or dark surface 
coupled with shallow engravings add to the difficulty. Thirdly, even if the content can be made out, 
any specific angle would only present a small section of the picture. To this purpose, even today the 
preferred method of viewing the seal in its entirety is by rolling out an impression on clay. This is of 
course highly invasive and in any case is not a viewing of the object in itself. 
In our tests seen here, RTI brought out incredible specular enhancement of a seal clay impression (See 
Fig. 4, left). Yet the seal itself yielded only partial information and even that far from perfect. Though 
highly detailed, the engraving was shallow, and the black stone did not respond well to raking light. 
The SFM, performed here on a different seal (BLMJ), while less detailed, offered the apparent ability 
to “roll” the seal itself in real time and manoeuvre the 3d light source to the best effect. This coupled 
with the Meshlab rendering filters, such as Lambertian Radiance Scaling and Ambient Occlusion, 
enabled stripping off the photographic texture map and concentrating solely on the surface texture. 
The Lambertian Radiance Scaling with Coloured Descriptor image (Meshlab) was brought into Adobe 
Photoshop where the RGB file was split into its three component channels. Each channel brought out 
other hitherto unseen details (See Fig. 4, right). 


RTI Cylinder Impression Meshlab Render - Radiance Scaling - Colored Descriptor 
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Figure 4: Seal impression. Israel Museum (left). Cylinder Seal. Bible Lands Museum. SFM. 
Meshlab Lambertian Radiance Scaling with Coloured Descriptor image and three separated channels 
in Adobe Photoshop (right).. 


5. RTI TO 3D AND OTHER ENHANCEMENT TECHNIQUES 


RTI and 3D representation of Cultural Heritage objects have their strengths and weaknesses; it is 
the combination of their relative strengths, which give these tools their usefulness in real-life 
situations. 

Attempts to convert RTI files to 3D format (PLY) or combine their relative merits have been made, 
such as the workflow suggested by L. Pawlowicz. Further work has been suggested by Elfarargy, 
Rizq, and Rashwan [2013], based on surface normals. 

Likewise, the Combination of 2D, 3D, and RTI Data has been demonstrated and proven effective on 
paintings [Aure, X. O’ Dowd, P. Padfield, J., 2017] 

Other techniques for the enhancement of our surface detail interpretation include ongoing projects, 
such as Algorithmic Rendering and the CARE (Collaborative Algorithmic Rendering Engine) tool in 


102 


joint development by CHI and Princeton University. 

It is not the purpose of this paper however to concentrates on the ongoing scientific research and 
development, but rather to suggest the empiric application of common and cheap or open source tools 
to the benefit of the heritage specialist. 


6. CONCLUSION 


The several test cases presented here, along with many more, attest to the assumption that the 
needs, contexts, and substances subject to surface detection and analysis, call for a variety of imaging 
procedures. It would seem that in many, if not in most cases, one technique would not suffice in 
providing all the information necessary nor certainly all the visual information possible. 

The two main tools, RTI and SFM, both have their strengths, and both are continually undergoing 
improvement. 

As with many tools, however, it is our practical experience with both their strengths and their 
limitations which can guide us in their use. Neither technique can yet offer us the complete answer to 
our needs. The field of surface detection demands proficiency in photography, in lighting, in RTI and 
3D construction, in image processing software, and above all in the willingness to harmonize, 
combine and assimilate these components in line with the demands of the task. 

As image resolutions increase, as algorithms improve and as software and hardware become more and 
more cost-effective and easy to use, it is fair to assume that we will witness exciting developments in 
the years to come and the stories, still hidden within the surfaces of our heritage objects will yet 
emerge and yield their secrets. 
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The architect of Chambord, the most famous castle of the Loire, remains an enigma. The 
absolutely innovative construction of this dynastic castle makes it a true marvel of the world, 
intended to immortalize his Majesty, the king-builder François ler. On September 6, 1519, 
the day he became twenty-five years old, Francois ler started the construction of the dungeon. 
From 1516 onwards, Leonardo da Vinci was the first painter, engineer and "architect" of the 
king, and housed by him at the manor of Clos Lucé d’ Amboise, but unfortunately he died on 
May 2, 1519, that is just before the construction began. 

The geometric design of a quatrefoil in Leonardo da Vinci's notes about the dividing of a 
circle and the quadrature, is our starting point. (Codex Atlanticus, Fol. 471, Biblioteca 
Ambrosiana, Pinacoteca de Milan). The circular towers of the castle give us the key. 
Measuring them we discovered that each tower is about 20 meters in diameter: 10 toises, 
which will be the module of the architectural plan. In those days, the masons used as 
measures the foot of the king (0.324 meters in length) and the toise (19.44 meters in length, or 
6 feet). The measuring instruments of the time are the compass (maximum opening of 6 feet) 
and the chain of surveying (10 toises), these are also the elements according to which will be 
drawn the plan and the different scales of the castle of Chambord. 

Our communication will outline the principle of the construction. A perfect set of proportions 
characterizes this ideal construction and coincides with the gold numbers of proportions. 


Could Leonardo da Vinci be the architect? 
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At the intersection of research and visual arts we have developed a method for visualizing 
archaeological hypotheses that explicitly preserves the scientific content excluding any unintentional 
content. It consists of virtual modeling and virtual photography. Other than usual we do not consider 
the spatial model as the decisive core but as an integral part of the visual mediation, and its 
complementary counterpart, the virtual photography, as equally important. Contrasting the usual way 
to reconstruct architecture in a hyper-realistic way as in the movie industries, we consider our 
approach a unique 3D Development and Application in the Cultural Heritage Area. 


INTRODUCTION 


Contrasting the geometric abstraction of the model, strictly based on the verbal hypotheses, our 
way of depicting the scenery uses traditional methods of realistic architectural photography. 
Leaving out any of this staffage, the emphasis of our approach lies on translating scientific 
hypotheses from text to image without leaving the scientific grounds and enabling they images not 
only to illustrate science internally among scientists as well as toward the greater audience, but 
enabling them also to mediate the timeless qualities of European architecture as shared patrimony, 
yet contributing to architecture as European Heritage. 


STATE OF THE ART 


The usual way of mediating architecture of the past via visualisations has been extensively 
developed my the movie industry and furthermore by the gaming industry. These visualisations are 
maximized in their engagement and in their capturing the spectator. Their primary if not only 
purpose is to give the spectator a feeling of diving into the scenery, of becoming even part of the 
scenery. This is achieved by an immense amount of purely ficticious additions. 


SUGGESTION 


Our approach emphasizes on the contrary the scientific content of the visualisation. Vividness is 
pursued as well but only in second line. Primary targets for our scientific visualisations are: 


e Their ability to serve as visual communication just as verbal hypotheses 


¢ Their ability to mediate to the public not only the scientific content itself but also its 
scientific character, that is, including all uncertainties. 
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e Their ability to mediate architectural concepts as timeless intellectual achievements, not 
any longer bound to the era of their creation but serving as eternal sources of 
architectural inspirations. 


The basis for this approach is the insight that also verbal hypotheses profit from their high degree 
of abstraction. Words are the most abstract means of communication. As just as verbal abstraction 
is highly if not uncompromisedly accepted, we suggest to use a lighter amount of this abstraction in 
visual communication means, too. 


Fig. 1 Trajaneum, Pergamon, around 200 AD 


Fig. 2 Skyline seen from opposite hill, Pergamon, around 200 AD 
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Verbal scientific hypotheses make use of the verbal abstractions and therefore there is a lot of 
uncertainty that remains unnoticed. But even if the uncertainty becomes explicit, the verbal 
habituation allows for vague descriptions and even contradictory alternatives. 


The use of abstraction in visual communication reveals a number of benefits: It allows the spatial 
evaluation, an exchange in another media and therefore in a broader sense and an exploration that 
leads to further questions and ongoing research. Furthermore, if realized in transdisciplinary teams, 
visualisations offer the possibility for other disciplines to pose their respective questions, too. 


“ceti 


Fig. 3 Alter Dom, Cologne, around 1025 
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Fig. 4 Cologne Cathedral, around 1540 
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The public benefits from the combines mediation of scientific content in an insight into scientific 
uncertainty. It perceives and understands that knowledge may be based on evidence, on analogies 
and on well-founded hypotheses, and that hypotheses can even be contradictory. It realizes that 
similarly to reading a text, looking at a scientific visualisation challenges the personal imagination. 
The visual perception is a concept, reality is a matter of mind. Architecture as a concept may be 
depicted, but reality remains a matter of imagination. So far this is fine for the primary sciences 
archaeology, building history and art history. But there is another benefit that arises from scientific 
visualisations. 
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Fig. 5 Domus Severiana, Palatine Palace, Rome, second century AD 


Fig. 6 Circus Maximus and Palatine Palace, Rome, second century AD 
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As scientific architectural visualisations are limited to the pure architectonic design concept, 
many other things are excluded that clearly connect with the past, especially the clothing of people, 
but also other everyday items. Leaving all this apart, the architectural concept of the spatial 
composition, the relationship of volumes and voids, the transgression in space, all of these are much 
more clearly perceptible. The supposed loss is actually a gain. The emphasis of the architectonic 
concept makes the concept timeless. The concept is capable of becoming a source of inspiration for 
today’s design processes. But it also clearly demonstrates the timeless quality and the common 
foundations of architects of all times. European Heritage as such becomes visible. 


Fig. 7 Interior of suggested ideal church by Julius Echter, Wiirzburg, seventeenth century 
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Fig. 8 Exterior of suggested ideal church by Julius Echter, Würzburg, seventeenth century 
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The challenge is to combine the abstraction of the verbal hypotheses with the necessary 
capability of the visualisation to engage and capture the spectator. This requires a coordinated set of 
parameters from the modeling to its projection. One major requirement is the coherence of the 
image. This means that every element is an integral part of the scenery. This cannot be achieved 
without compromises. But as we negotiate a lot with the primary scientists, we reach at least a state 
free from contradiction. The first step is to design abstract geometry that represents the hypothetical 
content together with its uncertainty. In the second step the abstract geometry is photographed as if 
it was real. Just like a photographer we search for suitable spots and illumination. We call this step 
„Virtual photography“. 


Fig. 9 3D model and VR of suggested ideal church by Julius Echter, Wiirzburg, seventeenth century 


If coherence defines the content of a visualisation, there are two major factors in photography 
that are responsible for a proper evaluation of the hypotheses. These are the unambiguous eye level 
and the unambiguous inclination of the view. On this basis, there is still an infinite range of design 
decisions. It is not trivial at all which time of day you take the photo at and which lense you use. 
The design quality of the photo determines the impact of the message. 


Moving pictures enrich the spatial experience. Cross-fading makes changes easer to notice and to 
compare building phases over time. Moving cameras reinforce the spatial impression. Physical 
models close the circle from the spatial hypothesis via the verbal and visual representation to a fully 
spatial experience. They allow an immediate vision as the spectator can freely move around the 
object. Except they are not suitable for real scale buildings. Real scale experience is where Virtual 
Reality becomes indispensable. The scales in VR are flexible, architecture is at least visually 
perceivable close to reality. The visual field is limited, the viewer looks as through diving goggles. 
Apart from the visual constraints, the major difference is the lack of real movement. Also, the 
spectator does not see his own body apart from stylized hands separated. This enables the user to 
interact with the virtual objects. But visually the technology is not yet mature. The resolution grid is 
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visible and the latency creates travel nausea at some users. As long as latency and resolution both 
do not match nature, VR is an experiment with advantages and disadvantages. Until this is resolved, 
a large perspective has its own qualities and “virtual photography” remains an important factor in 
the mediation of lost architecture. 


Proximity to scientific hypotheses always includes compromises. Especially the weighing of 
geometry against polychromy constantly demands hard decisions. The scientists’ priority has so far 
always been to abandon the very uncertain polychromy in favour of the comparatively certain 
geometry. This applies to nearly every situation in classic antiquity although the buildings of the 
time have generally been polychrome. 


Fig. 10 Cologne Cathedral, Choir, around 1850 


But colour is visually dominant, and because the colouring is in most cases so highly fictitious 
the geometry would nearly remain imperceptible. In later eras, the findings and therefore the 
knowledge becomes more certain, and in cases where the majority of the image parts are well 
knows in their particular polychromy, a scientific visualisation can be rendered polychrome under 
continuous consideration of the principles of proximity to the hypotheses. 


The project of the metropolis of Pergamon (Fig. 1 and 2, by Dominik Lengel and Catherine 
Toulouse, BTU Cottbus-Senftenberg) [1,2] was funded by and elaborated in cooperation with the 
German Research Fund Excellence Cluster TOPOI. Its last monographic exhibition took place in 
Leipzig in 2018 as part of Sharing Heritage, the European Cultural Heritage Year 2018 and so being 
literally a European Commission project regarding cultural heritage, including 2D — 3D Digital 
image acquisition for the re-contextualisation of 3D scanned sculptures in their hypothetical 
original architectonic context. Since 2008 scientists work with the geometric accuracy of this first 
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three-dimensional overall model of the metropolis. The model does not only evaluate spatial 
relations, it also reveals unforeseen visual phenomena. 


The project of Cologne Cathedral and its Predecessors (Fig. 3 and 4, by Lengyel Toulouse 
Architects, Berlin) [3] was funded by and elaborated in cooperation with the cathedral hut. It is 
being exhibited in the entrance area of the archaeological zone in the cathedral since 2011. The 
choir (Fig. 10, by Lengyel Toulouse Architects, Berlin) [4] was funded by the choir of Cologne 
Cathedral and elaborated with the archive of Cologne Cathedral. 


The project of the Palatine Palaces in Rome (Fig. 5 and 6, by Lengyel Toulouse Architects, 
Berlin) [5] was funded by and elaborated in cooperation with the German Archaeological Institute 
in Berlin and was first exhibited in the Pergamon Museum Berlin. 


The project of the Ideal Church of Julius Echter of Mespelbrunn (Fig. 7-9, by Lengyel Toulouse 
Architects, Berlin) [6,7] was funded by the the Cultural Foundation of the German Federal States 
and the Bavarian Savings Bank Foundation and elaborated in cooperation with the former building 
master of Cologne Cathedral, Prof. Dr. Barbara Schock-Werner. It was first exhibited in the Martin 
von Wagner Museum in the Würzburg Residenz exploring human-computer interaction for cultural 
heritage applications by combining flat visualisations with physical models, an auto-stereoscopic 
explanatory video and an Oculus VR experience. 
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Innovative methodologies and technologies. 
Civil liability and "genetic code" of a work of art 


Sara Penco 
Restorer and creator of Smarticon project 
Rome, Italy 
E mail: info@sapereproject.com 


Abstract 

The art world witnesses the path the evolution of mankind. Therefore its value derives not only from 
the conservation of the material of the work of art, but also from the recovery of the intangible heritage 
that determines its "reason to exist". 

Modern times impose an adjustment to the cultural heritage sector that lays the foundation for its 
valorization and safeguard, in order to become a resource with a continuous growth. 


Smarticon: a methodology to support the Cultural Heritage system 
Smarticon is a patent demand number 102015000006508. It is a “Method for the classification, cataloguing and 
tracking of valuable assets, in particular works of art”. 

Innovation arises from the awareness of the needs and the critical aspects of a sector and it takes shape 
through the identification of avant-garde solutions: that are able to solve problems, improve the state of 
the art and adapt to the needs and priorities imposed by the frenetic progress that characterizes modern 
times. 

The valorisation of an art work assumes the awareness of having to preserve and re-evaluate it 
according to two fundamental criteria: conservation, intended as the restoration of the material that 
forms the good and which is identified in the "material heritage"; but also the recovery of the 
cognitive heritage, which allows the "comprehension" of the meanings hidden behind the image and 
which represents the "intangible heritage" of an artefact. 

The heart of Smarticon is a complex database within which the information assets lavished by the 
experts (the source of knowledge), assumes a hermeneutical sorting and becomes easily accessible by 
the user. 

The method is cognitive and interrelates the characteristics of an artefact with all that information 
(currently without an order), suitable to trace the historical path and the origin of a good (published or 
unpublished). 

The first step consists in the recognition of the subject portrayed in the work of art (of any type and 
origin), through the identification of the iconographic attributes that characterize it. 

Once the represented subject is identified, the Smarticon system allows to study in deep the research 
(within the "subject record card"), to also identify a historical and geographical origin. 

The iconographic typology, in fact, over the centuries, has undergone frequent variations, which allow 
us today to establish real and true "rules" useful to identify the historical context from which the 
artefact originates. 

We can open a parenthesis. Think of the representation of the Crucifixion. Over the centuries it has 
undergone many iconographic variations, reflecting the dominant features of the thoughts and religious 
feelings of each period. For example, the Virgin, who was depicted standing next to the Cross until the 
Middle Ages, in the fifteenth century is now cowered on the ground because the artists emphasized her 
pain. The Council of Trent condemned this interpretation and re-established the iconographic "rules" 


116 


imposed by the texts of John the Evangelist: therefore, after the mid-sixteenth century, Mary was 
represented again standing. 

These iconographic characteristics represent a reasonably shareable and demonstrable resource for the 
purposes of "coding" a work of art. 

Other databases investigate the relevance of further information with the characteristics of the artwork 
under exam: the chemistry of pigments, ancient historical documents in which descriptions and/or 
ancient censuses are present, images of comparison between different types of manufacturing, old 
notary deeds, censuses in collections, publications, exhibitions, etc. 

Smarticon, through engineered and methodical processes, investigates the recovery of the intangible 
heritage of an asset, enhancing it both from a historical and from an economic point of view. 


“The noises of the world, of the weapons, of the conflicts strike me deeply. 
The severity of the crisis of society and of humanity have driven me to research at a radical level of 
theory, towards a progressive reorganization of the structure of knowledge” 
Edgar Morin 


The role of Smarticon and the blockchain in art 

The blockchain other than a technology, is a paradigm in which the register is structured in a series 
of blocks that contain the transactions and whose validation is a consent mechanism distributed on all 
the knots of the network authorized to participate in the process of validation of transactions 
(Https://www.blockchain4innovation.it/esperti/blockchain-perche-e-cosi-importante/). 


It allows the transfer of currencies, information and contracts in the framework of a decentralized 
infrastructure and accepts all types of digitization. 

The blockchain is based on five fundamental concepts: decentralization, transparency, security, 
immutability and consent. 

It is a "platform that allows the development and the concretization of a new form of democracy, 
really distributed and able to guarantee everyone the possibility to verify, to" control ", to have a total 
transparency on the deeds and decisions, that are recorded in immutable and shared archives that 
have the characteristic of being unalterable, unchangeable and therefore immune from corruption”. 
(https://www.blockchain4innovation.it/esperti/blockchain-perche-e-cosi-importante/). 

In the artistic sphere, the blockchain allows to equip the artefact of a code, within which all 
information relating to the asset is recorded (expertise, publications, diagnostic analyzes, transfer of 
ownership, prices, etc.). 

"The technological solutions defined as blockchains fall within the field of the Distributed Ledger 
Technologies, that means that it is about solutions based on Distributed Registers that allow the 
reading and modification by several subjects participating in the Network. The elements of 
differentiation between the various typologies of DLT are primarily in the modes in which we 
“govern” the control and the verification of the writing actions on the Register, on the modalities 
and the type of Consent necessary to validate these actions and on the structure of the distributed 
register itself.” (Article by Mauro Bellini: "Blockchain: what it is, how it works and the application 
areas in Italy"). 

The cognitive patrimony lavished through the processes of Smarticon, lends itself to become one of 
the "blocks" of the code of the blockchain and will assume a strategic valence because the recovered 
information derives from sources considered univocally certain, therefore they are to be assumed as 
"valid". 


117 


The goal is to obtain awareness of the work of art as a whole, since it is at the same time a 
material good and an intangible heritage. 

This process of enrichment, codification and preservation of the "genetic code" of each artefact of 
art will allow to reach new horizons of knowledge and sharing; but it will also assume a strategic role 
in terms of safeguard and security within the sphere of law enforcement. 

The recovery of this precious information represents an objective documentation also to support the 
subjective work of the experts, who are in any case entrusted with the task of definitively certifying the 
asset and, as an implicit and direct consequence, of certifying its value. 

The time is ripe so that all contributions (methodologies and technologies of various kinds) may 
converge towards a single tool, to safeguard one of the most important sectors of each civilization. 


Smarticon and machine learning 

Computer sciences and communication networks have become essential tools for the researchers of 
humanistic science. 

In the art world, the new challenge of information technology is to provide to industry insiders 
effective tools for data analysis, with the aim of facilitating their use and study and to promote digital 
heritage and cultural content accessible through archives, museums and libraries. 

In this context, artificial intelligence (AJ) is the tool to correctly process information, with the target 
of building and making available digital libraries that are easily accessible to any type of user. 

The integration of methods, services, systems and interoperability between different data structures, 
metadata and components, are key issues that have to be addressed by the scientific community to 
provide appropriate solutions to the needs of the sector. 

In this context, Smarticon's innovative contents lend themselves to support the community of scientists 
and researchers, offering them a patented method to collect and organize digital information from the 
Cultural Heritage sector in digital form. 

Aside from this peculiar knowledge base, Smarticon will be usable as a cloud software application 
that will exploit artificial intelligence to apply cognitive methodologies to support the user (scientist, 
researcher, tourist, occasional user) in tracking down and discovering unknown information and / or not 
obvious relationships on the work of art being investigated. 


The advancement of technology imposes to fulfil the need to be able to study the work of art 
through tools that allow to achieve a more focused and conscious comparison between the user and the 
artefact: be it a work of art, an architectural artefact or a complex system like an area of the city. 

All the IT procedures, of course, will always be supervised for approval by the experts, since the 
human experience remains irreplaceable. 

Artificial intelligence represents a strategic tool, since it harmonizes and enhances the functionality 
of innovative technologies, of the tools already available and of their effective application to the whole 
humanistic area. 

In this sense, Smarticon will act as a sort of interactive encyclopedia in the sector of Cultural 
Heritage and will be able to advance hypotheses, identifications, information, diagnoses, relations with 
other works of art that are distant (or close) in time and space. 

This "guardian" and "tutor" of culture, in the form of a software application, will make access to 
information much easier than an online encyclopedia and will obviously provide a much more vivid 
and profound cultural experience: an interaction and not a mere consultation will be created. 
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The management of Cultural Heritage. Innovative methodologies 


and technologies. Civil liability and "genetic code" of a work of art 

Today, numerous companies vigorously practice the so-called "civil liability" towards the company. 
For example the pharmaceutical market and food production, which provide basic needs to people. 

The Cultural Heritage also supplies the primary need for culture and civilization to society. 

It is therefore beyond doubt that those who manage art must do so with a civil, ethical (choice of 
correct behaviour) and deontological (with sense of duty and professionality) towards humanity. 

The world of art, besides being an inestimable cultural value, it has always represented an important 
resource to preserve and enhance: both for those who must administer it (museums, foundations, banks, 
etc.), and also for the antique market (auction houses, merchants, etc.) and for collectors. 

Over the centuries, the figure of art dealers has nourished thriving businesses that, often and still 
today, can be definitely more profitable than many other investments. 

The Cultural Heritage sector, in fact, through the course of recovering the preparatory information 
necessary for the enhancement of the asset, can generate enormous capital gains between the purchase 
price and the resale price on the market. 

However, a fundamental aspect must be emphasized: the opportunity to generate substantial profits 
corresponds to the danger of high investment risk. 

The authentication of the value of the good is "without question" subordinated to the consent of very 
few experts who represent the only figures of reference for the attribution of the artefact to a specific 
artist. 

If the work of art is not recognized, the investment is irremediably lost and without any possible 
valid alternative. 

A question has to be asked: how many times our investment in art will meet the favour of the 
experts responsible for approving and validating it? 

This mechanism, although acceptable given the characteristics of the sector, also represents a 
considerable deterrent at the expence of the "passion investment" and, moreover, it witnesses an aspect 
that makes it largely "foreign" and "independent" compared to the practices that characterize the other 
business sectors. 

The other market segments, in fact, have to adapt their strategies to the increasingly pressing need to 
comply with the "best practices" that influence both the longevity of a given business and the 
proposer's affirmation on the market. 


In this regard it is interesting to mention one of the stimulating articles published in the book by 
Danilo Broggi, an expert in economics and finance (“Diario di bordo. Tra innovazione e 
cambiamento” (“Logbook. Between innovation and changement”). Published by Guerini and 
Associates). 

L’autore, nel capitolo 52 intitolato “I (falso) oro del reno”, afferma: “...l’obiettivo di massimizzare 
il profitto senza rispettare dettami, norme, precetti e buone pratiche, esce dalla piattaforma del 
mercato e diventa mero capitalismo”. 

The author, in chapter 52 entitled "The (false) gold of the Rhine", states: "... the goal of maximizing 
profit without respecting dictates, norms, precepts and good practices, leaves the market platform and 
becomes mere capitalism". 

Broggi tells us how the company that invests the most in innovation in the world, the colossus 
Wolkswagen (the world's first company in the automotive sector), "... behaving like a capitalist, but 
breaking the rules of the game", "... consuming the emissions scandal ... "has prompted the market 
reaction that" ... came with unprecedented violence ...", generating a loss of ten billion dollars for the 


119 


value of the Wolkswagen brand, eighteen billion dollars for car collection and fines, the collapse of the 
worldwide sales and other consequences such as reputational damage. 

The book states that "... capitalism is unfinished without market behaviour ...". 

In the past, the field of Cultural Heritage has adapted to mechanisms which are too independent 
from the rules of the free market. 

Lacking a decisive push towards innovation and the sharing of the mechanisms that regulate this 
segment compared to the current models, the critical aspects of the sector have left room for scope for 
action with the violation of rules and good practices. They also ended up nourishing illegitimate traffic 
and crime, which have found fertile ground in a multiplicity of "loopholes" to elude controls. 

The police forces are still counteracting illegality without being able to count on the strategic 
support that could arise from the concept of a real shared "system". In this regard we will cover the 
topic in the next chapter. 

Another fundamental aspect, that testifies how innovation implies an effort to adapt, is the 
awareness that until a few years ago it did not seem ethical to do business through art. The mentality of 
the sector employees was so reluctant to the vision of Cultural Heritage as a source of economic 
resources, and also skeptical in thinking of profiting from it by planning business strategies (as if 
economic resources belittled the value of the asset or as if to preserve it, restore it and to enhance it was 
not necessary to put in place strategies aimed at finding the funds to then make investments). 

The efforts made have always been concentrated mostly on the enhancement of museum activities 
(which would benefit not negligible benefits from sharing knowledge by participating in the "market 
platform"). But today the mentality is changing. 

An important seed of innovation has sprouted a few years ago, when a new horizon was opened 
oriented towards collaboration between public and private sectors and overcoming bureaucratic 
obstacles in favor of communicating a fundamental concept to the world: the awareness of a shared 
vision of Heritage Cultural as a resource of humanity. 

I believe that this "partnership" has definitively consolidated with the financing of the businessman 
Diego Della Valle for the restoration of the Colosseum: an extraordinary enterprise of global 
resonance, which has switched on the spotlight on Tod's brand guaranteeing an enormous propaganda 
value on an scale international scale, with everything that with everything that follows. This operation 
has forever linked the name Tod's to one of the most important monuments, known and visited in the 
world! 

The merits of the businessman are to be admired, he who, with his determination, has resolved every 
bureaucratic obstacle and marked an epochal turning point by setting a concrete precedent in the 
convergence between the interests and the mentality of the public and the private. 

This strategy represents a striking example of how the relaunching of a company's production 
system can, at the same time, bring and benefit from the collaboration between two realities, two 
realities until recently independent one from the other. 

Even in the context of projects finalized in innovative methodologies and technologies, strategic 
synergies can be created between the public and the private sector for the benefit of both! 

Still quoting Broggi's book: "... Digital innovation can / must become one of the fundamental levers 
both to improve (and greatly) the relationship between the citizen and the administrative bureaucracy, 
and to support the relaunch of the productive system ... it takes ... new tools, new methods and, above 
all, a shared vision on the challenges to be faced and the ways with which intervene ... A systemic 
approach is needed...". 


Does a question arise spontaneously? To what extent the market segment dedicated to Cultural 


Heritage will wait to adapt its "best practices" to the physiological and frenetic technological evolution 
that characterizes our times? 
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Innovating means enhancing and safeguarding Cultural Heritage, meeting the sharing needs, 
adapting its survival according to the actual needs, as well as consolidating the strategic role that this 
sector can perform at a social and humanistic level throughout the world. 

It is necessary to integrate and adapt to the mechanisms that regulate the market, so that the 
economic resources necessary for investments can be produced; since there cannot be innovation 
without the use of economic resources. 

But not only. With the advent of the internet, it is becoming essential to adapt the "genetic make-up" 
of a work of art to the consent of a plurality of factors. 

There are many technologies and projects dedicated to the enhancement of Cultural Heritage and as 
many public and private efforts projected in this direction (for example, you can see the new 
indications of European directives). 

The mechanisms that regulate Cultural Heritage will not be able to avoid dealing with the rules of 
the global market, with new technologies and with the best practices for the growth of each sector and 
for free competition. 

Cultural heritage is a resource of humanity and, for this reason, the need for "sharing" is 
increasingly pressing ... and "sharing" can only be planned through the use of innovative 
methodologies and digital technologies. 

The target is to create a system that makes interaction, comparison, monitoring and control of one of 
the most neuralgic sectors on a global scale possible. 


The world of art projected towards the traceability of goods and free 


competition 

The time is ripe for the art world to adapt to the laws of the market, harmonizing and aligning itself 
with the inevitable rules of best practice of every good business. 

In 1904 the first art found fund dedicated to art was established (La Peau de l'Ours). It had an 
extraordinary success thanks to the prudence and foresight of the promoter of the investments, who 
bought only a few works of the highest level and at affordable prices. 

Later a multiplicity of initiatives followed one another, but the only ones to achieve modest results 
are the "British Rail Pension Fund" (established by the English railways for the benefit of its 
employees in 1974 and which in any case had not success as the first), and the "Fine Art Fund”. 

The fact that both of these last two funds were linked to two important auction houses: respectively 
Sotheby’s and Christie’s, is not negligible. Namely the two realities that hold the "control" of the 
largest slice of the market segment on a global scale and, therefore, to be considered among the most 
expert and aware of the mechanisms of the market on a global scale. 

All other funds have not recorded great results. Many have even remained inoperative. Why? 

There are basically five factors that determine the success of an investment in art: 

- the first is inherent in the foresight and intellectual honesty of the promoter; 

- the second determinant element is in the state of conservation of the work of art, in its integrity 

and in a scrupulous and conscious restoration (material heritage); 

- the third and not insignificant limit of investments in art (at least for what concerns the Italian 

legislation), focuses on the ability to find valuable works of art accompanied by certificates of free 

and legitimate circulation, since the international market offers better opportunities to generate 
capital gains; 

- the fourth point is explicit in the ability to recover as much information as possible concerning the 

historical path of the work of art (immaterial heritage). Furthermore it is important that these data are 
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reliable, shareable and verifiable, in order to also support the study path of the experts in charge of 
certifying the good; 

- the fifth step is the certification of the good by the experts (appraisal, exhibitions, publications, 
etc.). 


Basically, the enhancement of the asset passes through the conservation of the material (material 
heritage) and the recovery of the "genetic code" of the work of art (immaterial heritage), for a better 
"guarantee" of authenticity and a prerequisite for its enhancement . 

Improving the quality and reliability of the offer means containing the risk of losses of both the 
cultural and economic value. 

Furthermore, protecting the capital of the investors, stimulates the growth of the demand and 
increases “passion investments” (which currently represent around 24% of the market) are increased. 


The innovative methodologies for Cultural Heritage (such as Smarticon), artificial intelligence and 
blockchain technology represent the tools that generate knowledge and awareness of the good and that 
provide the contributions for that innovation that is essential not only to adapt to the times but also, as 
Broggi observes, to fulfill the need to "... establish independent supervisory authorities with regulatory 
functions that, together, guarantee the application and respect of the rules, promote free competition 
and at last protect the weakest individuals to defend themselves who interact inside and outside the 
company areas ... ". 

These considerations, which represent the rules of good conduct in the other business sectors, 
assume an even more strategic value for the Cultural Heritage, as the particular critical aspects of the 
sector generate the even more vital need for greater regulation and transparency for the benefit, for 
example, of the fight against the illicit traffic of works of art. 

The Cultural Heritage sector needs a sustainable business model, which arises out from the idea of 
conceiving a scheme that generates the presuppositions for an open and free competition, which allows 
to adapt to modern times and to provide fair tools for a greater guarantee of the investments and 
investors; since "capitalism is unfinished without market behaviour". 

Greater monitoring of the ownership changes of the asset /origin and subsequent sales) and of the 
custody of the related information (appraisals, exhibitions, condition reports, etc.) is required. These 
data which accompany the artwork will have to be traceable, encrypted and recorded in immutable and 
shared archives that have the characteristic of being unalterable, unchangeable and therefore immune 
to distortive manipulations. 

This strategy represents an essential tool for the fight against crime and a deterrent for the illicit 
market and for inexperienced collectors (especially if, living perhaps abroad, they think they can evade 
controls with more ease). 

In the area of security, the aspect of monitoring the free movement of goods should not be neglected 
either, which is another neuralgic point of this sector. 

Last but not least, money laundering and terrorism are also nourished by investments in art, which 
are well suited to justify substantial movements of money and buying and selling operations with very 
advantageous capital gains. 

Today innovative methodologies and technologies can create a greater synergy through their 
harmonization and give life to a real "system of Cultural Heritage", in which each component is also 
monitored through the decentralization, transparency, security, the immutability and consent that 
govern the mechanisms of the blockchain. 

The benefits would not be negligible and the effects would have positive consequences on the whole 
system. 
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In 2005, a first version of the current Smarticon method has been specifically designed to 
facilitate the identification and recovery of stolen works and has been provided to the Carabinieri 
Corps for the Protection of Cultural Heritage for the "Leonardo System”. 

The innovative contents have been the subject of commendations and awards (Site: 
www.smarticon.it). 

It is hoped that the invaluable base of knowledge created by this first installation can be included as 
an integral part of the knowledge base of the new Smarticon, which implements the new technologies. 


"... art is the instrument by which man has written the history of his civilization... 

... the work of art is the expression of a refined culture, which will survive long after the death of its 

creator... in timeless space without borders... 
. our artistic heritage represents the most precious good, allowing mankind to become aware of its 

origins and of the evolution that led us to be what we are today”. 

Sara Penco 
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This paper presents Ukrainian inter-university educational and scientific project on the 
introduction of information technologies to the restoration of works of art. The approach of 
increasing IT usage in the processes of conservation and restoration of artworks, and the use of 
data science and IoT methods in particular is considered. Further research ways and the project 
development directions are determined. 


INTRODUCTION 


Today IT supports all the spheres of life and conservation and restoration of art works should 
not be the exception. The coherent Ukrainian project working group for the introduction of IT to 
the field of art works conservation and restoration consists of the representatives of three 
universities: Taras Shevchenko National University of Kyiv, National Academy of Visual Arts 
and Architecture, Kyiv and State University of Telecommunications, Kyiv. The project objective 
is to improve the information-analytical, educational and practical aspects of conservation and 
restoration of art works based on the implementation of IT. 


Traditional approaches to the conservation and restoration processes of art works can be 
characterized by the high influence of the human factor, the difficulty of the physical and 
mathematical description, and the considerable amount of data processed [1]. In addition, the 
preciousness and uniqueness of art works requires adequate decisions regarding the current state 
of a particular monument, the choice of methods of conservation and restoration and 
comparative estimation of the cost of conservation and restoration, localization and analysis of 
losses, late records and layers etc., and also the provision of the possibility of preliminary 
visualization of the expected result. 


Not less important is the need to create appropriate microclimatic conditions for both 
conservation and restoration processes of art works and subsequent storage of monuments. This 
requires the use of modern IT such as data mining, machine learning, AI, IoT etc. The 
implementation of these technologies for solving the above-mentioned tasks of conservation and 
restoration of art works is what the activities of the project participants are devoted to. At the 
moment, the main areas of research within the project are identified, and being discussed below. 


The informational and analytical part of our work includes the systematization of restoration 
passports and creation of a search engine, by standard, unified items, as well as according to 
special characteristics of the state of preservation of a particular artwork, restoration processes, 
used restorative materials and chemicals. Such a system makes it possible to combine the 
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exhibits with similar technics and technological characteristics, to monitor and analyse the 
restoration processes and the results of the restoration measures taken. The significant base of 
photo fixation of exhibits (before, during and after restoration) in different types of lighting has 
existed since the foundation of the Department of Technology and Restoration of Works of Art 
of the National Academy of Fine Arts and Architecture in 1969. Consequently, this is a 
remarkable visual material that is also a subject to systematization, analysis and management of 
data. 


The necessary component of our project is the expansion of the educational process features. 
The systematization of the search for information in the restoration passports opens up the 
possibility of connecting the points of search of the restoration measures taken to the curricula 
component parts of the discipline "Restoration of Easel Painting" (the author of the program T.R. 
Tymchenko). Also, this systematization of the search can be used for visual preparation for 
practical questions in students’ tests. Such a systematic search is a unique unpublished database 
for students, postgraduates, lecturers, and scholars. 


APPROACHES 


To improve the modernization of the informational learning environment at the Department of 
Technology and Restoration of Works of Art of the National Academy of Fine Arts and 
Architecture, the project group is working on the creation of a unified data base for the 
accumulation, search and analysis of information. Such developments are relevant and will 
definitely serve to optimize the learning process. Ukraine already has successful examples of IT 
technologies implementation for the accumulation and systematization of information in the field 
of restoration [1-3]. Thus feature of our project is a multidirectional approach to the use of 
systematic information. The project organically combines informational and analytical, 
educational, methodological and research components. 


There are many opportunities for using IT in the restoration of works of art [4-6]. In our 
opinion one of the most promising is the use of Data Science methods. This is primarily due to 
the presence of a significant amount of data in the subject area of restoration of works of art, on 
the basis of which a number of studies are required. 


Consider the approach to applying data science methods in the restoration of works of art. The 
generally accepted algorithm for data analysis includes the following steps: 

1. Receiving data 

2. Preparation of data 

3. Model training 

4. Testing the accuracy of the model 

5. Obtaining and visualizing the results of simulation. 


At the first stage, digital projections of works of art are obtained from the results of various 
kinds of photography, like spectrographs, schematic images of various technological layers, 
stages of restoration processes, etc. 


Then, data is being prepared. The resulting set of projections requires pre-processing to bring 
them to a single size and resolution. After bringing all the projections received into a single 
coordinate system, one can consider the digital model of the original in the form of a set of 
points, each of which contains a vector of numerical characteristics obtained from a set of 
corresponding projections. That is, the digital model of the original is presented in the form of a 
multi-layered object, each of the layers is presented in the form of a matrix (array) of points with 
corresponding characteristics derived from the construction of a separate projection. Summarized 
this digital model is a data set. On the basis of the data set, it is possible to build a variety of 
models to obtain a certain result. As a result of pre-processing, a unified set of projections is 
created, which uses a variety of mathematical methods, and data mining methods in particular. 
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At the model training stage, according to the problem, various data science methods such as 
linear and nonlinear regression, classification, clusterization, etc. are used. According to the 
results of the stage, parameters of the model are obtained, which allow to determine the 
necessary parameters of the result on the basis of a digital model (for the given example - 
assigning model points to damaged areas of a work surface, which, in proportion with the total 
number of points, will determine the fraction of the damaged (lost) surface). 


To assess the accuracy of the model, its practical application for another set of data and the 
comparison of the results of simulation with the actual state of the work is carried out. When 
sufficient accuracy is achieved the model is considered suitable for further use. 


Consider the proposed approach to determine the total area of the damaged (lost) surface of 
the work of art based on the clustering method. According to the method, data with similar 
attributes is combined into separate clusters, among which it is possible to select clusters which 
include points from damaged sites of the work (see fig. 1). 


a) b) c) 
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Figure 1. Determination of the area of the lost parts of the art works on the basis of data 
science approaches (a — art fragment, b — fragment pre-processing, c — clustering into the base 
data set visualization, d — lost parts determination using data science approaches) 
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Figure la shows a photo of a fragment of art work with lost parts. After pre-processing the 
image, we obtain a photo with increased contrast of the intact and damaged sections (see fig. 1b). 
Applying the clustering method to this image, we obtain the initial set of data for further 
modelling (see fig.lc). In this set there is already a target areas distribution with the 
corresponding characteristics, which allows you to train the model and evaluate its reliability. 
After that, the model is used to determine the area of the lost sections of the whole work (see 
fig.1d). Similarly, it is possible to solve more complex image processing tasks to automate the 
restoration of works of art. 


Also it is promising to create information and learning environment for organizing the data 
base for accumulation, search and analysis of data that can be used to develop, control and 
improve the educational process. On the basis of web development technologies, databases and 
big data, it is planned to create an Internet platform that will serve as a powerful tool for working 
with the information obtained from the experience of carrying out conservation and restoration 
work in the past, at the moment, as well as the prospects of the field. The possibility of 
integrating this platform with other projects, including international ones is being taken into 
consideration. 


The research part of the project creates a wide array of practical applications. For example, for 
restorers, it has always been a difficult task to determine the volume loss of the base, the priming 
and the paint layer in relation to the total area of the picture, to determine the exact size of the 
losses and breakouts of canvas, and their coordinates (see fig. 2). The measurement process 
would take a lot of time and the result would be inaccurate. Application of IT technology 
significantly reduces the time and efforts involved in this process and gives more precise results. 


Figure 2. General view of the Ukrainian icons of the 18° — 19" centuries before restoration, 
authors unknown. 


Within the framework of the project, programs for analysing photo fixation in ultraviolet rays 
(see fig. 3) are being created. After all, the dark spots of overpainting, which are better shown in 
such a shooting, can be processed, amplified with the help of IT technologies, and impose on the 
photo taken in normal light. Such developments will facilitate the removal of overpainting. 


The roentgenographic research is obligatory for restoration of artworks, having several layers 


of different time overpainting [7]. Digital images which are made by using a computer 
tomography also give an idea of the lower layers of painting (see fig. 4). Creating a program for 
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overlaying a digital image of a computer tomography and a digital X-ray image into images in 
normal light is essential for understanding the layout of the lower layers of the image relative to 
the top layer and conducting other studies that are essential for the restoration of the exhibit. 


a) b) c) 


Figure 3. Ukrainian icon of the unknown author "Assumption of the Blessed Virgin Mary", 19" 
century. Bachelor’s Diploma on Restoration of Artworks by Antonina Mykolaichuk, 2018. 
General view of the icon before restoration (a), photophixation of visible luminescence in 
ultraviolet rays in the process of restoration (b) and after restoration (c). 


Figures 4. General view of Ukrainian icon "Christ Pantocrator", author unknown, the beginning 
of the 20" century and a snapshot of the computer tomography of the central part of the icon, on 
which we can observe the lower, author's layer of painting of the 18° century 


In the curriculum of the master's degree at the Department of Technology and Restoration of 
Works of Art of the National Academy of Fine Arts and Architecture there is a discipline 
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"Examination of Works of Fine Arts", the program of which provides a comprehensive 
examination of the artwork and writing of an expert-attribution study of a painting. 


Application of IT technologies for better reading of author's signatures and their counterfeits 
is already used for examination and attribution of artworks by the expert of painting Volodymyr 
Tsytovych [1]. Within our research is the introduction of computer programs, the use of different 
modes of detection of tone and colour contrast in order to improve the ability to identify half- 
effaced elements of the image and signatures. 


The issue of the reconstruction of lost elements of the image of artworks has always been 
debatable in the context of restorative ethics. The variability of computer reconstruction makes it 
possible to choose the most appropriate method of reconstruction and conservation for a specific 
artwork. 


Methods of computer reconstruction of lost parts of the image and signatures on the icons 
were described in the professional literature on restoration [8, 9], but are rarely used in practice. 
In our project, we explore the existing methods of image reconstruction, adapt them to a specific 
artworks and aim to make them more accessible to use. 


DISCUSSION 


Such an analysis will allow the virtual decomposition of art works, determine their current 
status as well as before and during the conservation and restoration, the degree and feasibility of 
restoration measures. Further it will allow to optimize the volume and sequence of conservation 
and restoration processes, and in the future, on the basis of IoT means, they could be partially 
automated. The subsequent development and improvement of these approaches to the projection 
materials analysis along with the accumulation of relevant experience will open the way to the 
use of machine learning and AI methods. 


Managing information and learning environment is crucial for organizing the data base for 
accumulation, search and analysis of data. This will result in developing, controlling and 
improving the educational process. To achieve these goals, the Internet platform will be created 
on the basis of web development technologies, databases and big data. This platform will serve 
as a powerful tool for working with the information obtained from the experience of carrying out 
conservation and restoration work in the past, at the moment, as well as the prospects of the 
field. Such a platform is viewed as one which can be integral part of other projects including 
international ones as well as evolving other projects in its development and upgrading. 


Moreover, one of the research directions of the project should be the use of modern IT 
facilities for the equipment of specialized premises for the conservation and restoration of art 
works. This direction involves the use of the capabilities of microcontrollers, sensors and other 
means of robotics to create separate elements with their further integration in complexes and 
networks. Among the main tasks of such complexes and networks are maintenance of the 
optimal microclimate of premises, robotic acquisition of projection materials, possible partial 
automation of the processes of conservation and restoration of art works. 


Finally, the preliminary assessment of the above-mentioned directions of research has made a 
program of the project, and helped to distribute tasks between the participants in order to begin a 
series of works for obtaining initial results. At the same time, a sufficiently important factor in 
the success of the project is the study and taking into account the best world experiences in the 
field. It should be noted that the project is open and does not exclude the possibility of expanding 
the membership. 


129 


SUMMARY 


Thus, the implementation of the project will ensure the creation of a powerful information 
technology surrounding in the field of conservation and restoration of art works, which will give 
an opportunity to provide support and maintenance of conservation and restoration processes, as 
well as their possible partial automation. It is also expected to intensify the scientific work of 
students, deepening the theoretical and strengthening the practical approaches of the educational 
process, further development of the universities cooperation. Project participants are interested in 
studying world experience and organization of international cooperation in the field of 
implementation of IT in conservation and restoration of art works. 
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Abstract: While museums have many ways to innovate, the dissemination of information and 
communication technologies (ICTs) in this field has recently received some attention for their influence 
and the possibilities those tools offer to enhance innovative strategies. Based on the existing literature, the 
paper presents a new typology of ICTs’ use in museums dividing technologies according to their 
characteristics, objective, and place of use. This paper investigates the creative engagement with digital 
3D models of museum artefacts and gives insight into new uses of museum collections enabled by digital 
scanning, editing and 3D printing technologies. 
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Introduction 


Museums and other cultural heritage institutions are one important part of the ‘creative industries’ and 
innovation in the creative industries provide major stimulating outputs towards economic evolution[1]. 
Innovation is essentially a social response to the changes in user behavior, however in 2 form of 
products/services designed based on the opinion and needs of the users. The active participation of users 
in today’s industries has revolutionized the way products and knowledge are produced, diffused, and 
consumed. This revolution generally relates to the technological revolution that happens in the 
information and communication technology industry. 

The new and social technologies have radically changed the way society functions from daily aspects 
to radical changes. The ICTs and other great inventions like the steam engine machine and electricity are 
compared with each other due to their ability of performing multiple functions through their practical and 
facilitating tools[2]. ICTs can serve as tools to enhance the way institutions use their internal operative 
systems, create revenue-generating activities, exhibit artifacts, communicate with its publics and the 
market, diffuse knowledge, conserve, and research. 


Current ICT is defined by three features. The first one, computational virtuality, corresponds to the 
infinite potential actualizations of an element implemented by the computer through bidirectional 
communication and logic -mathematical operations [3]. At a general museological level, the limitations of 
the physical dimension disappear, and it is possible to build any exhibition, by putting together the images 
of objects located in different times and places. 

The second feature, interactivity, has been defined as the nature of a system’s capacity to receive and 
respond to a human input action. Although having a slightly different meaning, this concept also exists in 
museums where it was introduced as part of the museological and pedagogical renovation of the second 
half of the twentieth century. Studies conducted in both formal and informal environments have proved 
that the active, self-controlled and collaborative exploration of digital contents indirectly benefits 
learning, especially in the case of complex, abstract or non-visible phenomena. 
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The third feature corresponds to the multiplicity of interfaces. In its current state of development, ICT 
can take various forms and serve different communication purposes. This is directly linked with the 
museographic level of the exhibition. Finally, augmented reality systems, based on wearable or mobile 
devices, are useful for exhibitions in which the object is still at the centre of the discourse, since they 
provide an additional layer of information about them which can be modified to set-up different kinds of 
visits. 

This paper presents the conclusions of an evaluative study conducted in the United Kingdom (UK) 
which brought three different contributions to the research field: it provided new empirical data about the 
effectiveness of high-tech exhibits; it tested a newly refined, specific evaluative methodology, and it 
changed the usual notion of ‘integration’ of ICT into exhibitions. A theoretical perspective might seem 
irrelevant in the design of high-tech exhibits, which are currently led by practical and technological 
aspects, but we agree with other authors that moving to a more abstract and general level is the key to 
understanding the relevant issues and enabling more effective solutions to be proposed. 

3D scanning technologies offer the opportunity to capture the form of real objects, and to store them as 
digital files. 3D print technologies are swiftly bridging the gap between digital and physical objects: it is 
now possible to print physical objects directly from digital files. 

Across the museum sector, research into the use of digital 3D technologies is focused on the 
implementation and support of core museum duties, including collection management, conservation, 
research and the interpretation of collections for the public [4]. A focus on knowledge-based and 
established museum practices risks ignoring more personal and subjective forms of museum experience, 
which can be explored through the use of digital technologies. To explore how creative engagement with 
digital 3D models can open up new forms of museum experience and new ways of understanding 
museum collections this paper examines two projects that promote the creative use of digital 3D models 
of museum artefacts, the (Im)material Artifacts project, undertaken by the researcher in collaboration with 
the National Museum Cardiff, and Lincoln 3D Scans, a project by the artist Oliver Laric in collaboration 
with the Usher Gallery and The Collection in Lincoln. 


Specific use of ICTs in museums: a new typology 


In the case of museums, the new technologies can improve and change institution’s operative and 
conceptual aspects. 

In pursuance of facilitating comprehension of their characteristics, divided the technologies addressed 
here into three main categories, in line with the innovative aspect they mostly correspond to, the 
theoretical review, and the empirical research material. It is important to highlight that this is a suggestion 
of typology of the technologies currently in use in museums, but it is true, likewise, that many of those 
technologies are multifunctional and are present, in different levels, among the other categories. 
Consequently, the suggestion of categories is: 

a) Informative and expositive technologies that enhance exhibition’s design and the presentation of 
artifacts/content, mainly during the visit (informative kiosks, audio and smart guides, interactive displays, 
3D, holography, virtual and augmented realities); 
b) Technologies that foster communication and marketing activities, promoting further and deeper 
engagement by the audience, mainly before and after the visit , and 
c) Technologies used mainly in organizational and managerial operations in the back-office of cultural 
institutions, in order to provide and combine the necessary elements for the exhibition and other more 
apparent technologies. 
Informative and expositive technologies at the exhibition level 

The most common on-site informative technologies found in museums are predominantly compost by 
services found at the exhibition areas such as plant introduction (PI) stations, personal digital assistants 
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(PDAs, or ‘pocket museum’), multimedia kiosks, dock systems, and audio and smart guides. The 
technologies at the service of the visitors in the hall or in the exhibition areas provide extra information, 
from the location plan to complementary information about the exhibition, permanent collection, and the 
institution. Info stations, kiosks, and docks were the first technological appliances at the service of 
visitors in museums[5]. Those appliances can offer from simple programmed information to joysticks, 
touch screens, and virtual reality tools. They are able to give basic information, enlarge understanding and 
contextualization of a piece, and to control equipment and systems(fig.1.). 


fig. 1.Diversified types of interactive technologies at KAUST Museum (http://www.kaust.edu.sa/). 


PDAs and audio/smart guides are mobile and wireless options to acquire extra information while 
wandering around the institution. These devices work as interlocutors for visitors in search of extra 
knowledge on the way and inspire interplay and stimulation. In addition, these devices offer extra services 
to visitors, such as audio descriptions for the vision-impaired public, and diversified mediums like videos, 
photos, sound, and texts. 

Visitors can also use technological tools as the quick response (QR) codes and applications in portable 
tablets and mobile phones, bringing the museum to the palm of people’s hands. QR codes give extra 
information about the museum or a specific exhibition through bi-dimensional barcode detection of an 
image. These codes give from basic information to direct links to applications and web pages. 

Applications also are tools that establish connection between the institution and the individual visitor, 
whom can download it directly to his/her personal mobile device and attain from practical information 
such as location and opening hours up to multimedia files and streaming [6]. Besides, one can customize 
it in accordance to his/her specific interests in a more dynamic and interactive way . 

Informative technologies are tools that serve as complementary forms to provide additional 
information. In the case of expositive technologies, however, they mostly refer to a particular piece, or 
determined part of an exhibition. Some examples of these technologies are interactive screens, 3D, 
holography, and virtual and augmented realities. They make use of diverse tools as images, illumination, 
sounds, and even mechanic vibrations to recreate, contextualize, and give the observer a full experience in 
addition to the attainment of information. Yet, the separation frequently happens into two types 
conforming to the experience they provide: immersive or non-immersive. Immersive technology makes 
the visitors feel similar to being in another time, space, and/or context through digitally or mechanically 
simulation of aspects of the world surrounding the artifact, where the public can interact using their 
bodies and movements; an example is the augmented reality technology. 
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3D and holography 
Tridimensional (3D) objects are geometric models created based on the real context where it can 


reproduce width, length, and depth of the chosen object. N Holography is the bi-dimensional (2D) 
projection of light in space, which the human eye can see and understand as 3D projections without the 
use of special apparatus like 3D glasses [7]. 3D technologies are usually more costly than holography 
once they require the use of complementary equipment or devices for the visualization of digitally created 
dimensional objects. 


The virtual museum 
Virtual museums have core functions such as exhibition, communication, research, and conservation 


of the pieces. Customization, interactivity, and content are some of key words when speaking of virtual 
museums, in addition to the informative objectives of a webpage. 

Online museums should offer organic navigation interfaces for each and all kinds of publics and, 
simultaneously, provide interesting, participative, and almost realistic experiences through new and social 
technologies, engaging the public and adding value to institutions’ image. The more authentic the online 
experience reveals to be, higher are the chances to bring online visitors into the real museum. Below, the 
figure 2 shows a suggestion for an art museum ontology combining actors and objects, which is 
apparently simple but sometimes difficult to apply. However, its implementation would produce a better 
model for museum’s websites to combine objects, discourses, and information. 


responsible 
for 


created in 
Place | 


Art 
Object 


| 
Curator Painting | | Sculpture 


created by 


Fig. 2: virtual museum’s ontology (Bailey, 2010, p.36). 


Both real and virtual museums’ professionals have similar curatorial objectives when selecting and 
defining the most appropriated communication methods to the prospective publics, working in accordance 
with the varied mediums, publics, and institutional goals. The objective is to use the digital element to 
add extra value and make more interesting online versions or even to create exhibitions available 
exclusively online. Conclusively, virtual museums should not become simply a digital replica of the 
physical museum but its extension, collaborating and cooperating to achieve institutions’ mission. 
Therefore, virtual museums, as with the real museums, should rethink its attributes and functionality, 
repeatedly, as the environment change itself nonstop. 

Information and communication technology (ICT) applications in UK museums 

In the era of the Information Society, where a new paradigm of knowledge construction and 
transmission is being built, static, unidirectional, massmedia containing text and images are widely giving 
way to interactive and personalized media containing hypermedia information. 

Many museums in the UK and the USA have long been using interaction as a means of interpretation; 
this trend had historically taken the form of hands-on activities [8] which now, with the advent of the 
technological era, incorporate computers’ capacities in order to create displays enabling a full new range 
of communicational possibilities . 
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According to our analysis, the degree of ICT integration in exhibitions seemed to depend on three 
main factors, including the knowledge domain, the exhibit’s specific communication goal and the age of 
the design. However, the conversations with the staff members corrected or fine-tuned the previous 
conclusions, providing a list of factors arranged not by their ‘summative’ importance, but by their more 
general, theoretical or more immediate and practical influence on the exhibition design. 

One of the primary factors involved in the perception and general use of ICT in exhibitions is, more 
than the knowledge domain, the role attributed to objects. As previously mentioned, museum 
professionals consider that technology is aimed at enhancing the understanding of objects: in the case of 
science museums, by providing the dynamic aspects; in the case of history museums, by providing the 
context and in the case of art museums, by providing an interpretation. 

The interviews confirmed that the second influential factor is the exhibit’s specific communication 
goal. This determines the chosen interface and justifies why we find similar interfaces with similar uses 
across all museums. It is considered the most intuitive, as it allows access to the WWW and can deliver 
different kinds of content: database-like multimedia, game-like multimedia and virtual reconstructions. 

The age of the exhibition had been considered to be the third influencing factor -the older an exhibition, 
the less integrated the ICT. As a result, they take ICT into account from the very beginning of the 
exhibition’s conception. 

The fourth influential factor is the awareness of the available technology. Choices are mainly made 
according to the content to be conveyed, but sometimes successful applications in other museums - i.e. 
mobile or immersive displays - can provide ideas for exhibits. 

Effectiveness of information and communication technology (ICT) in exhibitions 

The evaluation aimed to verify if the criteria that guided the design of the analyzed exhibitions were 
successful. To that end assessed those aspects that can be empirically evaluated: relationship with other 
exhibits and with objects, effectiveness with regard to the exhibit’s communication goal, exploration in 
group, usability, target audience and engagement. 

The weak relationship of ICT applications with other exhibits and with objects the interpretation of 
which they were supposed to enhance- appeared in all museums, regardless of their subject, the kind of 
exhibit or its goal. In the first case at IWMN, our survey and previous surveys showed that the spatial 
arrangement and contents of both high- and low tech exhibits, which were the majority, interrupted the 
linear and progressive thread of the visit and, at times, made it difficult for visitors to follow it. At MoSI, 
very few visitors interacted with high-tech exhibits in the way expected by the designers, because the 
different elements within the exhibition were not clearly articulated in a coherent spatial and conceptual 
discourse [9]. 

In the case of objects, observations confirmed the results of the evaluations conducted at MAG [10], 
which showed that, contrary to the main purpose of the interactive devices, visitors were not 
systematically linking the art works with them. The reasons for this are several: first, there are no 
identifying labels close to the art works; second, the mediators occupy a spatial prominent position within 
the whole exhibit and this situates the art work in a marginal position and finally, the activity is not 
always directly or clearly linked with the work’s meaning and consequently visitors did not understand its 
message. 

In general, high-tech exhibits achieved the goals for which they had been designed, and always related 
to the specific capacities of ICT. At MAG, families appreciated that they were invited to explore art 
works in an engaging and entertaining way, thanks to the interactivity and self-control provided by high- 
tech interactive devices. 

Whenever ICT failed to serve its purpose, it was not due to the technological component but to a 
problem in the overall design of the exhibition. For example, at IWMN, the visitor surveys revealed there 
was not a sufficient spatial and conceptual relationship between objects and the ‘action stations’. 

At MoSI (Figure 3), the summative evaluation concluded that either the gallery’s goal was not evident, 
or visitors did not retain the specific contents because of the structure and the communication strategies 
used by the external company, which came from modern design rather than from museography. 
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Group exploration was undermined in all exhibitions by the small size of screens and the 
computational, one-to-one communication paradigm of ICT applications. 

This problem is especially relevant in UK museums, where families are the main visitors. Large, 
passive screens (at MoSI and IWMN) accommodated several viewers well, even if they were not visiting 
together. But touch screens only allowed one user and one observer. This had two consequences. First, as 
reported in previous studies, family groups adopted different exploration strategies, including splitting up 
if there was more than one adult, turn-taking of children (at IWMN and MAG) and male visitors adopting 
the leading role as mediators. 

Second, inter-group communication happened mainly at hands-on exhibits because the observation at 
screens requires a physical proximity that can only happen amongst co-visitors. 


Fig. 3. The success of the exhibit was compromised because its goal was not clear at the 
Museum of Science and Industry of Manchester (MoSI). 


Since all exhibitions use touch screens, they should show very similar results. However, the 
differences revealed by surveys demonstrate that it is not only a matter of interface, but also of the 
position inside the spatial structure of the exhibition, the behavior expected by the museum inside the 
gallery, the visitors’ attitudes towards technology and the composition of the group. Moreover, the 
comparison of exhibits at IWMN and at the Science Museum showed that some ‘hands-on’ were 
designed, or located in such a way that they could only be explored by one visitor, and groups adapted 
their strategy exactly as described for ICT applications. 

“Poaching” the museum artifact 

Freely available photogrammetry [11] software has enabled museum visitors to create 3D models of 
museum artifacts using their digital cameras or mobile phones; “access to cheap, flexible tools removes 
many of the barriers to trying new things”. In Māori belief and in other cultural belief systems the 
affective properties possessed by real artifacts are sometimes seen to “inhere in their digital surrogates” . 
Freely available photogrammetry software creates a possibility for museum artifacts to be digitally 
appropriated. When this happens to culturally sensitive materials it might be considered inappropriate or 
offensive by parties with a vested cultural interest. At the same time, it brings new opportunities to the 
engagement with museum artifacts. 

Internet communities with the focus of sharing photogrammetric models are emerging and online 
collections house a diverse range of digital 3D models, including 3D models of historical artifacts from 
museums [12]. The creation and collection of photogrammetric models is emerging as a new hobby; it has 
the potential to change our understanding because it changes our relationship with objects. 

(IM)Material artifacts 

For this research the (Im)material Artifacts project was undertaken in collaboration with the National 

Museum Cardiff. A number of artifacts from the ceramics collections at the National Museum Cardiff 
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were selected for 3D scanning, and the resulting digital 3D models were made accessible to participating 
artists online. Participants were recruited online to remix the digital scans. 

The project culminated in an exhibition at the National Museum Cardiff, during which the participants’ 
screen based and 3D printed submissions were displayed with the original artifacts . 

Lincoln 3Dscans 

The Lincoln 3D Scans project by the artist Oliver Laric in collaboration with the Usher Gallery and 
The Collection in Lincoln was also investigated as part of this research. The project started in 2012, when 
Laric was invited by the Usher Gallery and The Collection in Lincoln to propose an idea for the 
Contemporary Art Society’s Annual Award for museums. Laric’s proposal to 3D scan the museum 
collections and subsequently publish all data for free was chosen as the winning project. 

This proposal led to the creation of the Lincoln 3D Scans website [13], where 3D models can be 
downloaded as STL files in order to be used without copyright restrictions. The Lincoln 3D Scans website 
includes a ‘gallery’, where the public can share images of the artworks they have created from the 3D 
scans. Users of the website are invited to treat the digital 3D models as starting points for new works and 
have the possibility of sharing their creative responses to the 3D models via the online gallery. 

Artworks 

Overarching themes emerge from the analysis of (Im)material Artifacts and Lincoln 3D Scans. The 
chief personal use for digital 3D models of heritage artifacts, created by users or accessed online, is 3D 
printing. 

Digital 3D models of heritage artifacts can be edited and personalized. With the necessary editing skill 
users are able to turn digital 3D models of heritage artifacts into ‘souvenirs’ of their own real and 
imagined experiences. Jason Rouse’s video game Postcards from Mexico, for example, blends the artist’s 
memories of Mexico with the digital 3D model of a Teotihuacan artifact . 

Postcards from Mexico was created for (Im)material Artifacts, a playable version of the game is 
available for download online [14]. The game transforms the 3D scan of a Mexican mask into a navigable 
virtual island. 

National Museum Cardiff his engagement with its digital reproduction was a form of repatriation of 
the piece to its country and culture of origin. Since the archives at the National Museum Cardiff provided 
scarce information on the background of the piece Padilla undertook his own research. He approached 
experts from the National Museum in Mexico, and discovered that the mask is likely to have been the 
head of a Teotihuacan figurine, rather than a ‘mask’, as it is described in the museum archives. 


Discussion 


During the projects discussed in this paper artists explored their relationship to the past, present and 
future, personal memories and associations and the relationships between technology and culture through 
the creation of derivative works based on digital 3D reproductions of museum artifacts. They also 
explored personal memories, narratives and associations through the creative transformation of digital 
replicas of museum objects. Participants poached and recontextualized the digital 3D models. 

The creative transformation of digital heritage communicates this ever-changing nature of our 
interpretation of the past. It reveals the importance of fantasy and the imagination and encourages a 
questioning stance towards circulated images and narratives. 

The trajectory of the digital 3D models used for (Im)material Artifacts and Lincoln 3D Scans has 
continued to extend online beyond the scope of the respective projects. 3D models from both projects 
were re-shared via online repositories. 

Digital 3D imaging and 3D printing technologies, especially DIY (Do It Yourself) and open source 
tools, are strongly tied into digital maker culture and the free culture movement. Today, the digital maker 
movement is a digital technology oriented extension of DIY culture and has roots in the Arts and Crafts 
movement of the late nineteenth and early twentieth century. 


138 


Creative engagement with digital 3D models of museum artifacts can promote digital literacy. On 
online 3D editing and printing theme boards and 3D file sharing websites users are able to share 
resources, tips and to communicate. When users begin to interact with each other ‘communities of 
practice’ can form; these can be viewed as social learning systems. They provide a “shared repertoire of 
communal resources — language routines, sensibilities, artifacts, tools, stories, styles, etc.” [15]. 


Conclusion 


In this thesis, explored the field of cultural heritage institutions, more specifically the case of 
museums, in order to identify the use and influence of the ICTs in respect to diverse types of innovation. 
The examination of innovation occurred through study of the creative industries due to the lack of 
material about innovation precisely in museums. 

One of the services of museums has long been to provide artists with rich material to inspire their art 
making. Digital 3D technologies have potential to support this function of museums. Through the creation 
and release of digital 3D models from their collections, museums can continue their role as sources of 
artistic inspiration in a digital arena. 

3D models from (Im)material Artifacts and Lincoln 3D Scans continue to be downloaded, remixed, 
printed and shared, and thus continually engage users with digital heritage and promoting increased 
understanding both of the original historical artifacts and the digital materials. 
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Fast development of interactive learning technologies needs a special consideration in a 
higher music education context. The new ways of learning content presentation through 3D 
imagining and through the theory of graphs application in music harmony are presented in the 
article. The proposed methodology is programmatically implemented in a series of web and 
Android applications, embodying additional game patterns for a better engagement of students 
into a learning process. A series of web applications has been successfully tested in educational 
experiments held with students in France and Poland, each time showing the high efficiency of 
the proposed methods and learning strategies. 


INTRODUCTION 


Fast development of technologies changed the knowledge paradigms. The attempt to apply 
cognitive models to the artificial intelligence development results in a reverse assimilation of 
the artificial network’s information circulation models by humans. The information society 
led to the emergence of the hypertextual logic, which became a new logic of narration in the 
21st century [1]. The problem of the information conversion into nonverbal quantitative 
forms was pointed out by Jean-Francois Lyotard [2], who assumed that knowledge should be 
translated into quantities of information in order to be operational in new conditions; a partial 
resolution of above-mentioned problem provided a frame theory proposed by Marvin Minsky 
[3]. 

The origins of hypertextuality lie in neuroscience research on human brain, especially 
from the modes of a human neural network functionality, which was applied to the 
development of artificial intelligence in the beginning of cybernetics [4]. Not only the search 
for modes of interaction in such an artificial neural network, but also the search for 
structures, in which the information could circulate, led scientists to the application of a 
“frame” term, as a unit of information, and a “graph” as its graphical representation. 

The cognitive aspects of the information, data organization and frames emerged as a 
reflection of the technical challenges of servers? communication, database management 
through operation systems and flowcharts representing the hierarchy in program commands 
execution. A large amount of information creates a database, which requires the application 
of quantitative methods of search, and, as a consequence, needs nonverbal ways of 
knowledge presentation which is frame and graph from a cognitive perspective. Operation 
systems represent a part of the artificial intelligence and use the frame logic for programs 
execution, which can be expressed in a form of the graph or a flowchart, therefore the frame 
and the graph can also be viewed from the cybernetics perspective. Therefore, quantitative 
methods of search correspond to the numerical artificial intelligence logic and such 
nonverbal ways of knowledge presentation as graph based on the frame is one of the ground 
principles for the information flow in artificial intelligence systems. 


PEDAGOGICAL CONTEXT 


Along with a data structuration discourse, the increasingly augmented role of visualization 
influences modern pedagogy. The role of a teacher has changed: teacher is no longer the 
source of knowledge [5], since no human can accumulate the amount of information mostly 
freely available online. The teacher instead can propose the systematization of a learning 
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material by creation of the learning algorithms using modern interactive tools, including web 
and mobile learning games. 

The learning attitude changes as well: the externalization of memory, mentioned by 
Michel Serres [6], leads to the search for ways of data structuring and compact nonverbal 
knowledge presentations. New tendencies in a learning material presentation assume it to 
become more interactive in order to fit the needs of modern students. 

Finally, the 21st century students are accustomed to technology, multi-tasking, fun, 
graphics and the Internet [7], thus learning technologies should aspire for a better 
engagement of students into a learning process by using these specificities for development 
of the appropriate learning strategies. 


A NEW THEORETICAL SYSTEM IN MUSIC HARMONY 


The research in a domain of artificial intelligence is based on human cognitive models. One 
of such models is used to develop a semantic network — the future of the Internet, which will 
have to make the decisions based on accumulated and analyzed information from big data. 
Systematization and rearrangement of information in artificial intelligence systems goes by a 
frame filling, which represents the smallest unit of semantic information and can be 
represented visually in a form of the graph. Several graphs form graph trees, resulting in the 
most compact view, which allows representation of a large amount of information. 


The system of graphs 


As was mentioned above, the frame is a structured unit of information for different situations 
description. Unfilled frame is called “proto-frame”, a frame being filled in is called “exo- 
frame”. Let us build a proto-frame presenting music harmony knowledge, by taking a passing 
progression as a model, as it gives great possibilities of the elements variation (Fig.1). 
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Fig.1. Two types of a frame: a) proto-frame b) exo-frame. 


In the received graph structure the top and the bottom vertices contain data about seventh 
chords and their inversions (the beginning and the end of a passing progression), and three 
middle slots (edges) contain data about triads with their inversions (all possible passing 
chords between seventh chords in a given top and bottom vertices). 

On the base of this proto-frame let us build the graphs for the rest of passing 
progressions possible between the first and the second, the second and the third inversions of 
the seventh chord of the Ist degree of a diatonic scale. The ensemble of all passing 
progressions for the tonic seventh chord forms a horizontal triad of graphs (Fig.2): 


Figure 2. Horizontal triad of graphs. 


The degrees used in the middle slots of each of the graphs within the same horizontal triad 
are repeating: V-VII-II, VII-II-IV, II-TV-VI and are in relation of a third between them. Only 
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the inversion of a triad changes if the migration to the next graph in a horizontal triad occurs. 
For example, the II degree appeared in a ground state (Is) in the first graph, then it appeared 
in its first inversion (IIs) in the second graph, and finally in its third inversion (II4s) in the 
third graph. 

If we build such horizontal triads for the rest of the diatonic scale degrees (II, III, IV, 
V etc.), we receive a system, in which the placement of one horizontal triad below another 
horizontal triad will be determined by common passing triads, forming a vertical triad. 
Correlations between seventh chords of the vertical triad are guided by the same rule of a 
third interval (V-III-I etc.) as for passing chords inside each triad. Each top vertex of the 
graph can be connected through a common edge to each bottom vertex inside the vertical 
triad (the system will be used for the main activity of the Android-based solution and 
therefore shown later in the paper). The presented system has been successfully used in 
music learning [8] and music analysis [9]. 


The enrichment of the music notation-based visualization 


The problem of the information structuration is not the only challenge which appeared in the 
information society context. The development of new media augmented our sensibility to the 
visual part of any matter of our existence. We became used to pay attention to charts, 
graphics, to get the substantial part of information from accompanying visualization before 
any verbal interpretation. In this context a traditional music notation used in a learning 
process needs to be reconsidered to become more informative regarding such crucial 
parameters in music harmony learning as a function of the chord and its structure. Thus, the 
enrichment of the traditional music notation with a color and its intensity has been applied in 
order to reveal these two fundamental parameters of harmonic sequences. The color groups 
were used to mark functional belonging to either dominant or subdominant group (Fig.3 a), 
and a color intensity was applied to highlight the structure of a chord in an open position. The 
decrease in the color intensity goes from the most important chord tones — fundamental and 
quint, to less important tones — third and seventh, accordingly (Fig.3 b). 
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Figure 3. Color and its intensity mapping to the music function and chord construction: a) 
mapping application to triads; b) mapping application to seventh chords. 


FRAMEWORKS FOR PRACTICAL IMPLEMENTATION 
Web-based framework 


The application of a new system of graphs to the learning process in a high school was done 
through the series of animated harmonic sequences visualizations and the interactive web 
applications. The developed interactive on-line activities for learning music harmony as well 
as created dynamic visualizations of the harmonic sequences represent two different 
pedagogical methodological approaches — the activity-based approach and the association- 
based approach. The association-based approach is widely used in the off-line education; 
however, the application of this method in the on-line education requires a large number of 
visualizations in order to be efficient. The activity-based method derives from the research of 
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Jean Piaget [10], which assumes that external activity causes an increase in mental and 
intellectual development, because the activity transforms not only the external object, but 
also the subject executing this activity. 

The animated harmonic sequences visualizations contained the described color and 
color intensity mapping to mark the chord function and the importance of tons within the 
chord structure and allowed highlighting the moving tones. The animation was also applied 
to harmonic sequences in a graph representation, reflecting occurring musical processes in 
the accompanying sound track, synchronized with animated visualizations. 

The interactive web applications consist of two types of learning activity. The first 
type comprehends the interactive graph filling by dragging and dropping the chord item: if 
the chord item was dropped correctly, the sound of this chord is played and the window of a 
whole 3D animated visualization of the sequence appears (Fig.4). 
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a) b) 
Figure 4. Screenshot of the first type of interactive on-line application. a) Example with a 
horizontal triad element; b) Example with a vertical triad element. 


This type of application differs by the form according to the use of either horizontal or 
vertical triad of graphs. The main goal of this activity is to provide a general view on a 
diversity of harmonization possibilities reflected in a graph representation. 

The second type of activity comprises deeper work with tones and intervals forming 
the chords of the sequence. The 3D notes’ visualizations showed in the graph part were the 
main material for work within this activity. When the cursor is placed on the pair of either 
two upper tons (top interval) or two bottom tons (bottom interval), the sound of this interval, 
as well as the degrees of the scale on which these tones are placed appear. After the first step 
of the chord intervals recognition, the second step follows, which consists of the similar top- 
bottom interval division of the chord but applied to the two last chords of the sequence 
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Figure 5. Screenshot presenting the second type of web applications. 
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The main goal of this activity is to show the basis of harmonization variation, assuming, that 
the same degree can be harmonized with different chords within the graph and can play 
different role in the chord structure representing different tones. The special task of the 
second step of this activity is to turn attention on the voices leading within the sequences. 

In created web applications three basic game concepts have been used, such as 
finding a right match concept in the graph activity (also used in a card game), discovering a 
new property in the notes activity (used in Black Jack card game) or receiving a bonus after 
the right match has been found (a basic game motivation) in the graph activity, as the 
animation video of the concerned harmonic progression appears after a correctly dropped 
chord item. 


Android-based framework 


The extended possibilities of interaction with the learning material provide modern mobile 
operation systems (OS). Such features as touch technology, the change of view with the 
device rotation provided by Android OS can be used for methodological purposes [11]. In 
order to explore these possibilities using the system of graphs in music harmony, an Android- 
based game has been created. The main activity screen proposes 7 horizontal triads to choose 
(Fig.6). After the horizontal triad from the main menu is selected, a new activity starts 
proposing the interaction with each graph of a chosen triad. 
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Fig 6. Screenshots of the Android game “Jeu d’harmonie”: main activity view. 


User needs to touch the edges of the graph to be able to listen to the sequence; the vertices 
and a chosen edge are highlighted showing the sequence played (Fig.8 a). The black panel in 
the bottom of the screen contains the passing chords of the corresponding graph presented as 
items with an interrogation sign inside. When the item is touched the passing chord is played 
and user needs to recognize the sequence from which this chord has been taken. If the item 
has been dropped correctly, the background behind the graph becomes green and user 
receives one point (Fig.7 b). In case of the wrong match, the background color becomes red 
and no score is granted. 
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Fig 7. Screenshot of a) highlighted chords while music progression execution, b) correctly 
dropped first passing chord and dropping of a second passing chord. 


The horizontal triad activity can be changed to a vertical triad activity with the device 
rotation either from the main activity switched to the vertical mode or directly from the 
horizontal triad activity. It should be noted, that the vertical triad which will appear as a 
result of the device rotation directly from the horizontal triad activity, will contain the graph 
from horizontal triad which user was interacting with before rotation. In such a way the 
methodological feature of the system consisting of horizontal and vertical triads is 
implemented: the device rotation results in a new activity (vertical, instead of horizontal), 
containing only one graph of the related horizontal triad, and not just another configuration 
of graphs from the horizontal triad. 

The functionality within the vertical triad activity resembles functionality described 
for the horizontal triad activity. However, instead of edges, representing passing chords, the 
bottom vertices representing the seventh chord inversions are given as a content of items 
which should be dropped accordingly in the bottom vertices slots (Fig.8 a, b). When all items 
are dropped, a pop-up window appears indicating the gained score and the actions to choose 
— repeat the activity or pass to the next activity (Fig.8 c). 


a) b) c) 

Fig 8. Screenshot of the vertical triad activity showing different states of the vertical triad 
activity: a) wrongly dropped fist seventh chord (bottom vertex of the graph), b) correctly 
dropped first seventh chord (third inversion of the dominant seventh chord: D2) and dropping 
of a second seventh chord (second inversion of the VII seventh chord: VII34), c) correctly 
dropped three seventh chords of the vertical triad and a score window. 
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PEDAGOGICAL EXPERIMENT 
Experiment conditions 


Student groups. The final expertise of the web applications efficiency has been 
investigated within the experiments held with 41 undergraduate students in musicology of the 
Michel de Montaigne Bordeaux 3 University (France) and later with 20 undergraduate 
students from the Institute of Music of Maria Curie-Sklodowska University in Lublin 
(Poland). 

Learning content. The amount of learning hours and the learning program was equal 
for control and experimental groups. Both groups worked in a computer classroom with the 
Moodle platform for distance education of Maria Curie-Sklodowska University, but in 
different on-line classes. The control group had the learning content organized as ordinary 
Moodle lessons with different pages. These pages contained described 3D animated 
visualizations, embedded as youtube videos with appropriate text description. Within the 
content of the lesson for the experimental group the links to pop-up windows were placed, 
leading to the described above interactive web applications. The experimental group had no 
text description in the proposed activities. The post-lessons tests were similar for both 
groups. 

Experiment results. All students had to pass two main tests: the pre-test and the 
post-test. During the course they also received the points for the post-lessons’ tests. The final 
estimation consisted of the overage estimation between final test estimation and a number of 
post-lessons’ tests estimations. Such a mixed estimation allowed measuring the engagement 
of students during the course. 


The impact of the interactive web applications on music harmony 
knowledge assimilation 


The students’ activity during the course was measured by estimations given for post-lessons 
tests. The measurements were done each two weeks, which was equal to one module within 
four lessons. The penalty for a delayed response consisted to 50% of the last received 
estimation. As mentioned above, the final estimation consisted of the overage taken from the 
final test estimation and three other estimations received for post-lessons test within each of 
three modules during the course, therefore the impact of the final examination was 25%. 

The charts below are based on the final grade, showing the overage estimation (Fig.9 
a) at the beginning and at the end of the experiment in both — the control group (marked with 
a red color) and in the experimental group (marked with the blue color). The estimation of 
quality of knowledge shows a proportional increase in both groups in the first experiment 
(Fig.9 b). Better engagement of students from the experimental group, impacted the overage 
estimation and the quality of knowledge in the second experiment (Fig.9 c, d). 
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Fig 9. The result of two experiments showing the estimations at the beginning and in 
the end of the experiment: a) first experiment: overage estimation; b) first experiment: 
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quality of knowledge; c) second experiment: overage estimation; d) second experiment: 
quality of knowledge. 


SUMMARIZATION 


The article presented possibilities of the learning process amelioration in a higher music 
education on example of the music harmony course. The appearance of such notions as 
“frame” and “graph” are explained as a result of the information society context and the 
proposed system of graphs in music harmony corresponds to the described phenomena. The 
article shows that the graph unit filled in with the information about chords, participating in a 
given sequence is a good alternative to the notation representation during the learning 
process, allowing the reduction of visual details and concentration on the phonic properties of 
the chord. Promising experimental results confirm the efficiency of the proposed methods 
application in a web framework with the use of basic game concepts. The created Android 
application for harmony learning presents a new edge of the subject development translating 
the problematics of the game-based music learning in a high school to mobile platforms. 
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ABSTRACT 

Volumetric imaging (true 3D image capture, as opposed to left/right 
stereo imaging) is being recognised as a new arena for recording images 
of the real world. Much of the current work in this arena investigates 
the technical aspects of volumetric moving image capture for Virtual 
Reality, often used for experimental immersive documentary and 
storytelling eg [Lgs This paper presents an alternative, hereto 
unrepresented area, discussing the visual aesthetics of volumetric still 
image capture for Virtual Reality, as a subject for experimental 
photography. Based upon the results of a long term (three years at time 
of writing) development of the Volca project [2, 3] for experimental 
volumetric still image cameras, this presentation discusses the various 
techniques for capture, their particular aesthetic qualities, 
opportunities and limitations, and compares the variety of aesthetic 
characteristics of various rendering, representation and dissemination 
processes and techniques for emerging area of experimental 3D imaging. 


INTRODUCTION 


The Volca [2, 3] project has been focused on development of experimental volumetric (true3d depth 
detection, as opposed to stereo- left/right depth simulation) cameras for still image portrait and street 
photography. Over the course of the project the technical development has investigated numerous 
depth sensors, capture hardware, software processes algorithms and visualisation techniques. The 
project has also explored in depth the challenges of disseminating and representing the depth-image 
content captured from the various iterations of hardware and software. Via this research through 
design process particular aesthetic trends and styles have emerged that through careful investigation 
and observation are discussed here and that this paper presents as an emerging aesthetic of 
volumetric photography. 


Principles Of Volumetric Photography 


Volumetric photography is based upon the recording of colour visual image data with per-pixel depth 
data, (sometimes referred to as RGBD data, [4] Red Green Blue colour and Depth) this depth data 
allows virtual reconstruction of a captured scene from a variety of angles with a wide degree of 
freedom. Comparatively stereo images such as the current slew of ‘3D’ movies in mainstream 
cinema rely on a limited simulation of depth in a chosen scene by providing a separated left/right 
pair of images (often viewed through coloured or polarised glasses) that give limited depth 
information from a pre-determined single viewpoint. 


Special hardware is commonly used to capture depth data in additional to colour visual data. This is 
predominantly visible light stereo pair cameras (such as the Stereolabs ZED camera [5]) where the 
left and right images are processed algorithmically to produce a disparity map [6] between the two 
images and therefore infer depth for individual points. Alternatively infra-red (the two main 
processes being structured light (eg the Occipital Structure sensor [7] where patterns of infra-red 
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light are emitted and then read back or time of flight (such as the Microsoft Kinect II sensor) where 
light is emitted and the time it takes to reflect off nearby objects is recorded) 


Fig 1. Kinect based DIY portable volumetric capture system 2018, Daniel Buzzo, sensor head 
and control hardware, Copyright the author. 


Each particular sensor method has implications for data quality and context usability [8]. The visible 
light stereo pair cameras have large effective range (commonly in excess of 15m) and work well in 
bright daylight, whereas infra-red sensing methods are limited in range (often less than 8m) and 
work poorly in daylight, conversely the infra-red sensors will work (at least regarding depth 
data) perfectly well in complete darkness where visible light sensing (and unfortunately the 
associated RGB image recording) can fail completely. 


Both visible light and infra-red techniques return depth values per pixel from a 2 dimensional raster 
array, up to the maximum x,y resolution of the respective sensors. (The Kinect | sensor is a 
structured light sensor and has a depth image resolution of 320 x 240 pixels and the Kinect 2 is a 
time of flight sensor with a higher depth resolution of 512 x 424 pixels. The Occipital Structure 
sensor is a structured light sensor with a maximum depth image resolution of 640 x 480. By contrast, 
the Stereolabs ZED stereo-pair depth camera has a stated maximum depth image resolution of 4416 
x 1242 pixels) They are also limited in their depth resolution, commonly becoming less accurate in 
their depth estimation the farther away an object is. (Structured light and time of flight infra-red 
sensors commonly have maximum effective ranges of 5-8m with accuracy of depth data diminishing 
with distance. By contrast the ZED stereo pair visible light camera has a stated range of up to 20m, 
though at maximum range the disparity map process this type of sensor utilizes become increasingly 
inaccurate). 


Depth sensing scenarios have a particular challenge compared with RGB photography in that objects 
beyond the sensing range of depth hardware are returned as null data [9]. Essentially creating ‘holes’ 
in the depth image - as opposed to areas of no light in RGB images (eg, shadows or night sky) it 
cannot necessarily be inferred what the null data in a depth image may be - it could be objects at 
great distance, or sensor failure, or equally nearby objects that are in ‘shadow’ from the infra-red 
emitter signal or sensor. The decision of how to represent this null data has the challenge that RGB 
image data for these null depth areas may exist and have useful subject data in. 

Representation of this null data has two basic options - either the data is discounted and the ‘data 
holes’ are translated into holes in a generated 3d surface (see figure 2) or alternatively the data is 
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‘back walled’ (see figure 3) - eg given an arbitrary maximum value that appears as a scenic, diorama 
style maximum value. Each has their own advantages and particular aesthetic qualities, and also 
representational challenges when viewing back pre-recorded RGBD images. 


Choices in volumetric capture 


Once depth sensing hardware is selected there are choices to be established regarding how data is 
processed and recorded from the hardware system. Depth sensors generally give depth per pixel (up 
to their maximum sensing range) across an x,y raster array. This depth data is most simply 
represented as individual points having x,y and z values. When viewed back as a 3d dimensional 
scene this data becomes what is referred to as a ‘point cloud’ (see figure 4) each point having 
arbitrary size and commonly coloured based upon the RGB value of an aligned colour image 
captured by a 2d visible light camera. Whilst this point cloud data is truest to what a depth camera is 
actually detecting other techniques can offer very different aesthetic interpretations of a captured 
scene. 


Fig 2. Wanchai Street market, 2018, Daniel Buzzo, volumetric street photography with meshed 
surface, Hong Kong, Copyright the author. 


Fig 3. Mother and toddler, Backwalled volumetric image, 2016, Daniel Buzzo, volumetric street 
photography, Hong Kong, Copyright the author. 


Translating the captured point values into surfaces requires skinning or meshing of the point cloud 
(commonly as contiguous triangles though sometimes as quadrilaterals) this also allows high 
resolution RGB data to be 'painted‘ as a texture onto the surfaces of the generated mesh. It also 
allows the generated surface mesh to be synthetically illuminated by virtual lights rendered in a 3D 
viewing environment. 


Decisions on how to treat null depth data (that now become visible as holes in the surface mesh) 
become important. Null data, possibly caused by objects beyond the range of the depth sensor, cause 
non-contiguous points in the point cloud to become joined during the mesh generation - sometimes 
causing triangles, or quadrilaterals, to stretch from very near to very far objects creating ‘false’ 
surfaces. Routines can be added to the mesh generation process to pre-clean and normalise point data 
and eliminate these false surfaces. Common routines also allow data interpolation to create interim 
data to remove ‘pinholes’ in surfaces and smooth them. 


152 


w i 
Fig 4. Man on Tram, point cloud rendered image, 2017, Daniel Buzzo, volumetric street 
photography, Amsterdam, Copyright the author. 


SLAM VS single shot 


There are also two significantly different approaches to the capture, processing and recording of 
depth data that greatly influence the results and aesthetics of subsequent volumetric images. Once a 
scene has been selected to be recorded, the data from the hardware can be sampled in two distinct 
ways. The first, and simplest is as a single ‘shot’ recorded over a brief (generally less than 0.1s) 
period. (see figure 3) In the same fashion as a visible light camera such as a DSLR has an exposure 
time a single shot volumetric camera will record depth and RGB data from a single view point over a 
short period of time. This has the advantage of being fast, allowing recording of moving or complex 
subjects, and giving control or greater choice of composition and intended viewpoint for the 
subsequent viewer. It has the necessary disadvantage that the depth data is only available for front, or 
sensor facing, data from the composition or subject. It also means that areas of the composition in the 
shadow of elements of the subject (from the infra-red emitters position) will have null depth data in 
the final rendering. Due to the nature of common hardware setups where the infra-red sensor and the 
RGB sensor are displaced from each other horizontally there can be parallax errors, i.e. discrepancies 
in registration between the apparent spatial positions of colour data and depth data, on objects close 
to the camera apparatus. This is commonly compensated for with calibration and registration 
corrections to align depth and colour data accurately. 


The second main process employed in capturing volumetric images utilise a technique from robot or 
autonomous vehicle navigation and mapping systems. The technique known as Simultaneous 
Location and Mapping (SLAM) [10, 11] is used to create and locate an observer viewpoint in an 
environment at the same time as generating a map of the environment. The SLAM technique samples 
data from a depth sensor and attempts to build a consistent 3D model of the surroundings over an 
extended period of time. [12] With each successive sample of depth data taken from a moving sensor 
the technique compares new data against stored data to create complex representations of 
environments beyond what can be ascertained from a single viewpoint. In practice this means that a 
volumetric camera will record depth information over an extended period of time, possibly many 
tens of seconds, as the sensor array is moved over and around the subject or scene. This allows 
capture of volumetric information of all sides of objects or of complete scenes. This has the 
advantage of being able to eliminate infra-red shadow occlusion and capture data of the sides and 
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backs of objects and scenes. It has the disadvantage of being suitable mostly for static or inert 
objects. Objects that move can create ripping or distortion effects where the SLAM algorithm 
attempts to reconcile data of a single surface in multiple positions. It can also cause disparity 
between captured RGB data used for texturing surface meshes and the shape and position of 
the generated meshes themselves. 


Image and subject choice and composition - depth, scale 
and aesthetic limitations 


Once RGBD data has been captured and processed the subject volumetric images can be recorded in 
a number of common 3d data formats. Common formats include the geometry definition format OBJ 
(or .obj) [13], originally developed by Wavefront Technologies, the format describes 3D geometry 
alone but can include Material Template Library (.MTL) files as a companion that describes surface 
shading (material) properties of objects. In the example of volumetric RGBD data the MTL file will 
principally reference an RGB image texture file captured by the RGB sensor with positional data for 
mapping of the photographic texture to the geometry described in the corresponding .obj file. The 
other common format in use is the Polygon File - PLY (.ply) format, also known as Stanford 
Triangle Format [14]. Each of these file formats describe the vertices or points in a volumetric image 
in x,y and z co-ordinates but also the relation of points that make up mesh surfaces. Such as lists of 
points at the corner of triangles making up surface meshes, their normals (a normal is a line 
perpendicular to the surface and is used in calculating how lighting affects a 3d graphic surface) their 
materials and the position of any texture image that may be part of the surface material. 


Surface representation techniques. point, mesh, surface, 
lighting - cleanup 

As noted above, depth data is originally generated, and captured as per pixel depth values creating 
point cloud information. These data points are commonly coloured via information extracted from 
the RGB data captured alongside the depth data. This colour information is necessarily limited to the 
resolution of the point cloud density. Viewers commonly infer contiguous surfaces from proximal 
points in a point cloud, making the ‘reading’ of dense point cloud representations relatively 
easy. From a view perspective, Point cloud representation also has the added benefit of not 
occluding rear or distant surfaces, often contributing to the viewer's sense of the depth and visual 
‘weight’ of the represented objects. When interacting with point clouds, the parallax movement of 
near and far (from the viewers perspective) points dramatically demonstrates the depth nature of 
volumetric images and makes a strong distinction, style wise, between captured volumetric images of 
real world objects and computer graphic (CGI) style generated models, such as animated 3d game 
characters and the like. 


Fig 5. Arex - portrait, 2018, Daniel Buzzo, volumetric street photography with exposed vertices, 
Hong Kong, Copyright the author. 
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Fig 6. Sewing, 2018, Daniel Buzzo, volumetric portrait photography with exposed vertices, 
Belgium, Copyright the author. 


When approaching suitable visual rendering of volumetric data this comparison, or confusion, can be 
one of the greatest drawbacks in appreciation of volumetric photography as an art medium in its own 
right and as an extension of photography. This is in strong contrast to exemplar 3D animated models 
that an audience is often more familiar with from cinema and TV advertising CGI work and gaming 
characters. 

As we discussed, the alternative to using point cloud representation is to use a surface mesh, 
either triangle or quad divided. This creates contiguous faces that are commonly shaded from a 
virtual light source and textured with the RGB data captured alongside the depth data. 


This method of surface representation allows more RGB texture data to be utilised as the mesh faces, 
being mathematical vectors between points, are rendered at the base resolution of whatever display 
medium is being used. Generally these mesh surfaces create occlusion of back surfaces and can 
sometime reduce the sense of depth and weight in a rendered scene compared to point cloud 
rendering. Surface rendered models can also run the risk, as discussed previously, of appearing as 
CGI-like renderings rather than real-world representations. A technique used to good effect in the 
recent exhibition of volumetric portraits in VR is to use surface rendering and accentuate the 
underlying mesh and vertex data, (see figures 5 and 6) deliberately illustrating the faceted nature of 
the captured data. This deliberate exposure of the relatively crude nature of approximations of real 
world subjects helps differentiate the volumetric images from modeled, game-style 3d objects and 
also engages viewers with the capture process, and by inference the captured nature of the images, 
directly situating the viewer with the original context of capture. In the same way that a 2D 
photograph can transport a viewer to the situation and subject of the photographic image rather than 
viewing the surface of the image as an image only. This area digs deep into the much wider arena of 
image, representation, surfaces and media theory and sadly is beyond the scope of this specific paper, 
it being directly related to volumetric aesthetics rather than the place of volumetric imaging in the 
philosophy of photography. 


Image dissemination and viewing 

Once captured, processed, meshed, surfaced, textured, cleaned and saved our attention comes to 
viewing and dissemination of volumetric images. Whilst the world of volumetric photography is in 
its infancy and it is difficult to say with any certainty how audiences of the future may use images, 
and how this may in turn affect the design of the apparatus (as Vilem Flusser [15] would term any 
kind of photographic imaging equipment [16]). Currently there are three principal methods of 
viewing volumetric images. 


1) interactive 2D displays such as laptop or desktop computers 

2) lenticular or light field 2D displays (including pseudo holographic desktop or ‘fishtank' displays) 
such as the 3D spherical displays of Fels et al [17] and the recent developments in light field displays 
such as Looking Glass [18] 

3) VR 3d headsets 


Each of these has their own particular sets of advantages and disadvantages and the context of 
viewing scenarios can greatly influence the audience's reaction and expectations. Exhibiting 
volumetric portraiture in VR can be beset with problems base on the pre-conceptions of the medium. 
At the exhibition of Volca (ibid) volumetric portraiture in VR in Montreal in spring 2018 a small 
number of viewers were confused by the contemplative nature of viewing only, asking what the 
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volumetric portraits ‘did’ and what they were supposed to ‘click on’ in expectation of a game-like 
interaction scenario. 


Particular promise has been shown in using desktop ‘fishtank’ style pseudo holographic displays [19] 
that allow headset free viewing of images with a string visual depth component. This style of display 
has a sense of pace and presence akin to viewing artist photographic books or collections. Giving the 
viewer a stronger sense of control and contemplation of the image as a complete element - 
not requiring additional interactivity or animation. 


Given the parameters and constraints and opportunities discussed the previous sections, several 
aesthetic choices can be made regarding subject, composition, capture process and subsequent render 
and viewing conditions. As one gains experience in the various factors involved in the origination, 
capture, manipulation, rendering, viewing and dissemination of volumetric photographs the influence 
of subject choice and scene composition became apparent. Because of the nature of depth sensor 
efficacy and accuracy over medium to long distances (20m or more) it is difficult to establish scale 
and context that in traditional photography buildings or landscape may give to a scene. Therefore 
small intimate observations appear more effective, particularly where there are people or there are 
easily recognised (and cognitively ‘sizeable’, commonly recognisable objects such as bicycles, 
coffee mugs, fruit etc) elements. In this way some parts of the current forms of volumetric 
photography may share elements with Renaissance oil painting, particularly the practice of still-life 
painting and posed portraiture. 


Discussion and conclusion 


Volumetric photography is in its infancy and what examples there are extant and commonly from the 
world of experimental creative technology rather than photographic development where practitioners 
are building their own hardware and software systems with which to experiment. However, as both 
hardware, such as depth sensors and portable computer systems, and software techniques become 
more understood and readily available there is a growing body of knowledge and co-incidentally a 
growing body of images accessible. The renewed interest in Virtual Reality and the parallel 
developments in pseudo-holographic displays may find that mass volumetric photography available 
to and generated and consumed by a wider public will be a significant medium in the near to mid 
future. 
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Abstract - Digital Cultural Heritage can often take great advantage from 3D 
reconstructions. 3D Web can be considered as a great opportunity for communicating this kind 
of contents, both as reconstructions relevant in themselves or as a tridimensional portal towards 
databases and other cultural resources. Often cultural environments are full of details, even 
long commemorative texts, as in the case of the Santo Stefano Lapidarium. In these 
eventualities, a good management of the 3D assets and a smart optimisation are of relevance. 


INTRODUCTION 


3D online cultural virtual environments are particularly effective to present 
themselves. This task can be achieved using 3D reconstructions of historical sites or 3D 
interfaces enabling the retrieval and access to further contents, such as multimedia data, 
documents or other cultural resources [1,2,3,4]. An example of 3D online environment for 
accessing data was performed at VisitLab Cineca (Visual Information Technology 
Laboratory) in 2004 for the project “Certosa Virtual Museum in Bologna”, with the virtual 
reconstructions of some commemorative monuments linked to a very large database 
dedicated to the First and Second World War and the Neoclassical third cloister of the 
Certosa cemetery [5]. 
In order to easily reuse and update the 3D environments and ensure cross-media outputs, 
Visit Lab has shifted in time towards an Open Source approach, adopting a pipeline based on 
Blender (www.blender.org) and WebGL libraries. In recent years 3D online deployments 
have been delivered in Blend4Web (https://www.blend4web.com/), as in the case of the 
MUVI project (Virtual Museum of Daily Life in XX century - https://muvi.cineca.it) - with 
the 3D reconstruction of a house of the 1930’s ported from the Virtual Theatre space, a 
Cineca’s semi-immersive facility where these 3D models can be virtually navigated, to a 3D 
online application [6]. A new 3D interface taking advantage of 3DWeb is under construction 
also for the Horizon 2020 I-Media-Cities project (https://imediacities.eu/) [7]. In this case a 
modular and adaptable 3D environment, called Virtual Collection Creator, will host and 
exhibit selections of audiovisual contents, coming from the thematic “collections” previously 
created by archives and researchers during their own investigations, automatically adapting 
its exhibiting virtual rooms to the number of items to be displayed. 
In any case, the need of optimising the virtual environments in order to be effectively 
explored online is a permanent challenge, with a direct correlation to the complexity of the 
cultural environment to be reproduced. The recent release by the Museum of the 
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Risorgimento in Bologna of a web application for navigating the 3D models of the 
Lapidarium in St. Stefano Bologna, a memorial dedicated to the dead soldiers of the Great 
War, can be considered as a possible example of specific difficulties to be overcome when 
opening online a virtual environment with a huge number of elements to be visualised. In this 
case, the application presents the 2536 names of the soldiers that were readable on the 64 
commemorative plaques of the Lapidarium and that are now, in some cases, washed-up and 
unreadable. 

The paper will present the technical solutions adopted in order to improve the management of 
the various elements of the application. 


WORLD WAR I LAPIDARIUM DEDICATED TO THE BOLOGNESE 
SOLDIERS 


The Lapidarium project is an evolution of the Certosa Virtual Museum, whose aim 
was to promote the knowledge of the Monumental Cemetery in Bologna by using it as an 
open air museum, telling the history of the city through its monuments and graves. The 
concept laying behind Certosa Virtual Museum was that of the so called “Talking 
Monuments”, that consists in making monuments capable of telling all the stories celebrated 
or mourned through them. Three locations of the Monumental Cemetery were chosen for an 
in depth in three different historical topics: the two monumental ossuaries dedicated to the 
First and Second World War and the neoclassical tombs, with their relevance for the history 
of art, of the third cloister. Each point of interest was transformed in a 3D model and 
connected to databases about people, places and historical events. At present, the online 3D 
models, used as a portal towards the three historical scenarios and deployed at the time with 
Exhibit 3D, a commercial plug-in, for technical obsolescence are not available on the 
project’s website anymore, while the database still exists on its own and is constantly 
enriched with new contents and multimedia resources by the Museum of Risorgimento in 
Bologna (http://www.storiaememoriadibologna.it/scenari). The current project about the 
Lapidarium tap into this same database. The Lapidarium, hosted in the cloister of St. Stefano 
church in Bologna, commemorates 2536 Bolognese soldiers listed on 64 commemorative 
plaques. The large database about the facts of the First World War in the Bolognese province 
contains, among its record, also all the biographies of the soldiers honored within this 
Lapidarium. In 2018 the existing 3D model of the St. Stefano cloister, created in SketchUp by 
the Museum of the Risorgimento in Bologna, was ported by VisitLab Cineca in Blender, 
adapted to online navigation using Blend4Web framework and linked to the database records 
(https://www.storiaememoriadibologna.it/lapidario-di-santo-stefano-in-3d). Through the 
application it is possible to switch between the cloister as it is nowadays and as it was in 


1925, the year when the memorial was opened to the public. The 1925 scenario allows to 
virtually visit the Lapidarium as it was originally conceived, with all the plaques in place. 
Navigating the version of the cloister as it is at present, a more effective contextualisation to 
the reconstruction is given, as the user can recognize the actual spaces of St. Stefano, with 
also some of the plaques removed in order to make space for the opening of some 
passageways in the walls (Fig. 1). 
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Fig. 1 - The World War I lapidarium: a view of the cloister hosting commemorative plaques 


Through the commemorative plaques themselves it is possible to access the database web 
pages. By clicking these objects, users enter a different navigation mode where they can both 
read the soldiers’ name inscribed on each plaque, due to the point of view located in front of 
it and the pan and zoom tools given, and they can find the link to the web pages related to the 
single plaque and the people commemorated by it. The user interface inside the “plaque’s 
inspection mode” (Fig. 2) reflects the importance given to the database links, since the side 
box containing them occupies a large portion of the screen. 

The virtual Lapidarium application aims at using a virtual reconstruction as a database’s 
records background: the 3D models become a sort of new user interface for the database 
itself. In this way the application can increase people’s interest in searching the database, 
since the investigation is made more engaging by setting some historical informations in the 
physical location where they come from. Links to already existing HTML pages coming from 
the digital database were used. Therefore, 3D Web enables new applications to take 
advantage of elements coming from previous phases of the projects and already available 
online. Since the main focus of the application is about the commemorative plaques, an 
excellent legibility for the engraved names is needed. The surface of the original plaques got 
ruined over time and, in some cases, the soldiers’ names became illegible. For this reason an 
ad hoc texture had to be created. The challenge was both in finding a way to have the highest 
possible texture resolution, given that the application had to be web navigated in real time, 
and of developing an efficient pipeline to insert in an automatic (or at least semi-automatic) 
way such a number of names in the 3D world. 
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Fig. 2 - Viewing a plaque in the “plaque’s inspection mode” 


MANAGING THE TEXTS 


As we already said, photos of the real commemorative plaques in the St. Stefano church 
could not be used to texture the virtual ones due to their current maintenance status. The low 
maintenance status of some plaques, in fact, led to the inadequate legibility of their engraved 
names. The appearance of the plaques in the current photos is also in contradiction with the 
time period setting of the virtual reconstruction, coinciding with the Lapidarium’s 
inauguration year, when, obviously, the newly set up plaques were still clean and legible. In 
addition to that, as we already said, some plaques were removed and got lost over time. As a 
consequence, their virtual reconstruction could not have been based on their photographs. 

In order to have a perfect legibility of the names, high resolution textures were needed but 
their use in a 3DWeb application for such a number of objects would have resulted in an 
excessive slowing down of the navigation. Indeed, a real time 3D Web application in order to 
be displayed at a proper frame rate by as many different devices as possible, cannot load too 
many heavy resources, such as high resolution textures. 

For these reasons the main aim in managing the plaques reconstruction was to look for the 
names best legibility possible, limiting the use and the size of textures, keeping in mind that 
3D geometry is lighter than raster images in terms of file size (Fig. 3). 
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Figure 3: A view of the walls of the cloister covered in commemorative plaques. Plaques are 
readable even outside the “inspection modality”. 


The adopted method is based on the use of a bitmap texture atlas, a technique commonly used 
for text rendering in the 3D gaming industry. In video games developing, when a bitmap 
texture atlas is used, all relevant characters and symbols from a chosen font are typed in a 
single large texture known as bitmap font. Each character, known as glyph, has a specific 
region of texture coordinates associated with it, and its size and location are described in a 
text data file. Every time a text is rendered, its specific constituent glyphs are selected via 
script (through a vertex array or VBO) and the corresponding bitmap font section is rendered 
to a 2D quad. 
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Fig. 4 - Example of planes making up the plaques texts and UV mapped on the bitmap font 
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Likewise, in our case a bitmap font is created and 3D planes, UV mapped on the texture atlas 
and placed on the 3D models of the different plaques, constitute the single letters of the 
engraved names (Fig. 4). This method has benefits in terms of optimization of both textures 
and geometry effects on the application final size file. Instead of one texture for each of the 
64 commemorative plaques, only two high resolution textures, the bitmap font for the golden 
letters of the names of the soldiers awarded with a gold medal of military valour and the one 
for the red letters of the other names, are used. Moreover, using planes for each letter instead 
of its mesh, made by lines and curves, means a considerable decrease in terms of 3D models 
number of vertices. While in video games texts are dynamically created in scene and 
rendered in real time as they usually have to change during the gameplay, in this case texts 
were created during the modeling phase and then exported in the real time asset. Coding was 
not used for text displaying, as usually is in video game implementation. It was used, instead, 
in order to make the pipeline for the texts creation more efficient. Python programming in 
Blender was used, in particular some addons developed by the Blender community and 
modified according to our needs. The steps of the texts creation pipeline were as follows: 

- Creation of the True Type Font (.ttf) file of the original chars actually used on the real 
plaques; 

- Automatic text formatting using a spreadsheet with all the texts from the original 
records with soldiers names commemorated in the Lapidarium. The text layout comes 
from the analysis of the real plaques and their spatial layout and contents 
arrangement; 

- Copy and paste of the layed out text in the 3D world using Blender internal text tool; 

- Bitmap font creation with a raster graphics editor; 

- Creation of planes, one for each bitmap font glyph, UV mapped on the same texture 
atlas previously created; 

- Transformation of texts from curves to geometry and text separation in different 
objects, one for each character, using Animation nodes (AN) addon; 
[https://github.com/JacquesLucke/animation nodes]. The addon was modified to 


name each object as the character constituting its geometry; 

- Replacement of the geometries of the single character with instances of the plane 
mesh containing the corresponding glyph. All the characters of the same type were 
selected and replaced at once taking advantage of the fact that, thanks to the AN 
addon modification, they had the same name. 

As all the single characters were replaced with instances of the same plane mesh of the 
respective glyph (thousands of objects which refers to only 33 planes, considering letters and 
special characters), texts can be easily editable in their properties as, for example, those of 
size, color, rotation or appearance. 

Therefore, as text displaying does not rely on coding, texts can be edited at any time also by 
people who have not programming skills, sach as museum curators. 

Since the application has a dissemination aim, the target audience has to be as wide as 
possible, as well as the kind of devices able to run it at a proper frame rate. For this reason, 
even if the technique presented in this paper has a very reduced effect in terms of increasing 
the final file size, in comparison with other methods that could have been used, further 
optimizations were needed to cope with the possibility of navigating the application with 
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devices with low performing graphic cards. 

The working copy was kept unoptimized in order to be easily editable, while in the models 
intended for the real-time navigation the number of scene objects was highly reduced by 
merging the different planes making up the single plaque texts into a single object, one for 
each different plaque. Level of Details were also created. All texts of the plaques set up on 
the same cloyster side were baked in a single low resolution texture. At a distance, where 
high resolution texts are no longer needed, the object made up by the single characters planes 
is replaced by a plain textured with the baked texts, speeding considerably up the rendering 
time. 


CONCLUSIONS 


An important question, when dealing with digital heritage simulations, is the one posed also 
by Champion: “Does the resulting simulation add a new perspective, which would be more 
difficult to design and deliver in other media?” [8] Furthermore, what does a virtual heritage 
app can offer that real resources cannot? In our case, for example, in comparison with the 
physical lapidarium, we have a database offering the sum of a huge quantity of information 
coming from different archive funds. Information that, otherwise, would be accessible only as 
single physical resources at the Museum of the Risorgimento. Besides, the engraved plaques 
are now readable thanks to the virtual restoration, and it is possible to peruse also those that 
have been stored in the warehouse or went lost in time. Hence, this application presents the 
reconstruction of the lapidarium as it is now, making it visitable by faraway people, and its 
original version, capable of joining the database to elements that are no longer visible. 

The economic limitations of this project forced us to find strategies to reduce the 
programming effort. The approach taken to deal with texts, presented in this paper, was not 
already provided by Blend4Web framework and shifted a lot of work from developers to 
artists, compressing the costs. Furthermore, in this way the most relevant aspect for the 
curators of the Museum of Risorgimento, that is the possibility of editing and correcting the 
names of the deceased, was kept feasible without the need to request for technical 
support/personnel. When the application was released the open codes were an opportunity for 
the Museum for modifying its aesthetics according to their taste (at present, therefore, the 
application looks different from our release presented in this paper). In future, they will have 
the possibility of creating further 3DWeb resources using other 3D reconstructions of 
commemorative monuments. 

The care used in making the names of the dead soldiers readable again is a way of paying 
them homage and preserve and consolidate their fading memory. 
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Abstract — This paper presents the final results of HERIBITS, a research project co-funded by the 
Tuscany Region Government, under the Operative Regional Program FESR POR 2014-2020. The 
project proposes an innovative methodology and a platform enabling bottom-up and top-down 
management of cultural heritage initiatives. The techno-social platform integrates collaborative tools 
for CrowdSourcing, analysis tools for rating project ideas and evaluate socio-economic impact, to 
propose best practices and to detect similar initiatives in order to avoid project duplications. The 
platform also integrates an ad-hoc CrowdFunding shop. 


SUMMARY 


Management of cultural project ideas and proposals in the framework of research and market 
oriented public funding programmes requires significant effort and resources, and faces several 
challenges: (i) a chronic inability in Public Administration to effectively engage and spend 
substantial community and national resources for underdeveloped areas; (ii) a lack of structured 
methodologies that provide the correct time to market of planned initiatives, enabling the alignment 
of projects to industry best practices and a coordination of cost centers, avoiding duplications and 
lack of critical mass; (iii) the need to improve the use of EU and national funds within the 
framework of regional development, for cultural heritage protection and valorization initiatives, as 


well as local identity promotion and cultural tourism. 


This paper presents the final results of the project HERIBITS, an ambitious research initiative co- 
funded by the industrial research program “Bando Unico della Ricerca” of the Tuscany Region. The 
project proposes an innovative method as well as advanced technologies enabling a new hybrid 
paradigm of bottom-up and top-down management of cultural heritage initiatives to improve 
collective awareness in the planning, sourcing and execution of Cultural Heritage initiatives and to 


optimize the allocation of public funds. 


The project developed a “techno-social” web platform supporting the entire process of evaluating 
cultural project proposals and ideas from selection, to planning, implementation and management of 
results, adopting a collaborative approach, to support public administration and their 
technostructures in the process of selection, design and implementation of a large scale of initiatives 


of cultural interest. 
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End users of the platform are citizens desiring to promote their cultural ideas (bottom-up approach), 
while customers B2B are public administrations in need to optimizes their mechanisms of allocation 


of public funding to cultural initiatives (top-down approach). 


The techno-social platform integrates collaborative tools for CrowdSourcing, as well as analysis 
tools for rating project ideas and evaluate socio-economic impact, using the emerging Logical 
Framework Analysis (LFA) method and integrating a third party analysis tool, Asset from K4D, for 


socio-economic statistical analysis dedicated to public administrations in Italy. 


The platform proposes best practices based on project matchmaking techniques, also enabling to 
detect similar initiatives in order to avoid project duplications. The platform provides also social 
network capabilities and integrates an ad-hoc CrowdFunding shop, integrating the public 
contribution for exploitation activities to ensure sustainability of cultural projects, especially after 
project conclusion. 


After developing and testing the software platform with alfa and beta users the project ended with a 


validation phase involving cultural heritage offices in Italian local public administrations. 


Business planning based on the emerging Lean Startup approach completed the project activities by 
devising a Business Model for the Heribits initiative. After a better definition of user and customers 
requirements by means of a field research during the first part of the project, we pivoted our 
Business Model from B2C to B2B, finally targeting local public administrations. The 
CrowdFunding emerged as a key aspect of the process of project proposals evaluation. A success of 
a CrowdFunding campaign for a project idea increases, in fact, the probability of success of the 
prospective project, as it shows commitment and interests of potential customers, in line with the 
Minimum Viable Product and Continuous Deployment concepts proposed by the Lean Startup 
approach adopted. 
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